Augmented%20Virtual%20Environments%20(AVE):%20Dynamic%20Event%20Visualization - PowerPoint PPT Presentation

View by Category
About This Presentation



Augmented Virtual Environments AVE: Dynamic Event Visualization – PowerPoint PPT presentation

Number of Views:36
Avg rating:3.0/5.0
Slides: 21
Provided by: ulrichn2


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Augmented%20Virtual%20Environments%20(AVE):%20Dynamic%20Event%20Visualization

Augmented Virtual Environments (AVE) Dynamic
Event Visualization
  • Ulrich Neumann Suya You
  • Integrated Media Systems Center
  • University of Southern California
  • September 2003

Problem Statement
  • Imagine dozens of video/data streams from people,
    UAVs, and robot sensors distributed and moving
    through a scene
  • Problem visualization as separate
    streams/images provides no integration of
    information, no high-level scene comprehension,
    and obstructs collaboration

A Simple Example USC Campus
Visualization as separate streams provides no
integration of information, no high-level scene
comprehension, and obstructs collaboration
AVE Fusion of 2D Video 3D Model
  • VE captures only a snapshot of the real world,
    therefore lacks any representation of dynamic
    events and activities occurring in the scene
  • AVE Approach uses sensor models and 3D models
    of the scene to integrate dynamic video/image
    data from different sources
  • Visualize all data in a single context to
    maximize collaboration and comprehension of the
  • Address dynamic visualization and change

Research Highlights Progress
Algorithm research and tech barriers inherent in
AVE system
  • Interactive Modeling System
  • semi-automated feature finding
  • linear/non-linear element fitting
  • ARMY TEC tech-transfer over summer 04
  • ICT tech-transfer over summer 04 and extension
    for rapid modeling underway
  • Video Capture System
  • up to 8 real-time internet video streams at
    704x480 at 12Hz with graceful degradation
  • Rendering System
  • real-time GPU code on dual CPU PC
  • integrated camera calibration system
  • texture retention and background modeling
  • interactive GUI and remote control for
    integration with Northrop Grumman
  • Image Analysis System
  • detection and tracking of moving objects (people
    and vehicles)
  • dynamic creation and placement of pseudo-models
    in 3D scene
  • rapid interactive modeling of object models

Integrated Modeling System based on
interactive fitting
Video Capture System
  • AxisCam 2120 (and similar) 704x480 image size
  • Three potential bottlenecks
  • (i) Ethernet bandwidth (gt20 cameras for 100bT)
  • 43 Kb per image at high quality compression
  • 12 fps camera image rates
  • bandwidth per camera is 12438 KB/s 4.13 MB/s.
  • (ii) MJPEG decompress time (8-10 cameras --
    current limit)
  • 100 fps per CPU with Intel IP library (MMX, 3D)
    for IDCT
  • 8 streams _at_ 12 fps consumes one CPU
  • MPEG-2 cameras are coming (expect gt20 cameras)
  • (iii) Graphics system bus bandwidth (gt150
  • AGP 8x and PCI-Express are fast

Rendering and Application Control
  • High performance rendering with real-time GPU
    code on dual CPU PC
  • Network video and XML interface for integration
    with existing sensor networks and monitoring
  • Views automatically zoom to alarms,
    geo-referenced positions, or user selected views
  • Alarm status icons reflect site status
  • Patrol-mode automatically flies user-defined
    path(s) over the entire site
  • Integrated camera calibration system
  • Local and/or remote user(s) control system via
    joystick, keyboard, and mouse
  • Customizable control and interface modules and
    SDK in development

Video Processing
Dynamic Event Analysis
  • Dynamic object detection
  • Background estimation
  • A variable-length temporal averaging algorithm
  • Adaptive histogram thresholding
  • foreground object segmentation
  • Morphological filtering noise rejection
  • Object tracking
  • Optimal matching between current objects and
  • Consider both spatial and temporal coherence
  • Matching criterions overlap, size, motion

Dynamic Modeling
  • Object modeling
  • dynamic polygon model to fit segmented
  • planar or 3D generic models
  • Object placement
  • 3D parameters (position, orientation and size)
    based on
  • ground assumption
  • object class constraints (e.g. vehicles have 4
  • path smoothing

Interactive Rapid Modeling (initial
Modeling from single (and multiple images) by
fitting to parameterized base models What can be
done in a minute?
Tracking and Modeling Results
Video Texture Management
  • Problem Video texture projection paints the
    scene each frame
  • Object movement changes visibility/occlusion
  • Camera movement paints new areas
  • Dynamic visualization control (viewpoint, image
    inclusion, blending, and projection parameters)
    reveals unseen areas
  • Texture management provides more complete view
    of the scene
  • Texture retention shows recently seen data
  • currently occluded by moving object
  • currently unseen by moving camera

Video Texture Retention
  • Challenges
  • Accumulation of video textures leads to infinite
    texture storage and system slowdown
  • Merging of multiple textures requires accurate 3D
    sensor calibration
  • Varied image-to-object mapping between frames,
    illumination, resolution etc
  • A Model Based Approach
  • a base-texture-buffer for groups of model
  • warp each new image to the base-buffer via sensor
    model and 3D model
  • visibility/occlusion test via depth map
  • rendering by traditional texture mapping with the
    base textures - fast

Texture Refinement
  • Artifacts arise in reconstructed textures from
    tracking errors
  • Use 2D image registration in base-texture buffer
  • affine motion model for image registration
  • feature (corners) based matching
  • least squares solution for warp-parameter

Texture Management Results

no texture management
with texture management
Northrop Application Collaboration
  • Military base surveillance situational
  • Different sensors (CCD/IR cameras, motion
    detectors, radars) deployed throughout the test
  • Networking and XML interface communicate with the
    sensor network and a main control station
  • System monitors and responds to sensor status and
    alarms, providing overall site visualization
  • Views automatically zoom to alarms,
    geo-referenced pre-sets, or user controlled views
  • Patrol mode automatically flies user-defined
    path(s) over the entire site

Surveillance Scenario
Future Plan
  • Tracking of PTZ Cameras real-time tracking
    using model-based vision and calibrated mounts
  • Dynamic Modeling real-time tracking and
    model-fitting for moving objects
  • External, in-line processing
  • Texture Management real time texture retention,
    progressive refinement
  • System Architecture - scalable video streams and
    rendering capability, (PC clusters?)

  • Collaboration with Northrop Grumman
  • Install v.1 system (8/03) for demonstrations
  • Install v.2 system (9/04) for demonstrations and
    evaluation license
  • Tech transfer
  • Source code for LiDAR modeling to ARMY TEC labs
  • LiDAR modeling and AVE visualization integration
    into ICT training applications for MOUT
    after-action review
  • Publications
  • IEEE CGA Approaches to Large-Scale Urban
  • PRESENCE Visualizing Reality in an Augmented
    Virtual Environment
  • IEEE CGA Augmented Virtual Environments for
    Visualization of Dynamic Imagery
  • CGGM03 Urban Site Modeling From LiDAR
  • VR2003 Augmented Virtual Environments (AVE)
    Dynamic Fusion of Imagery and 3D Models
  • SIGMM03 3D Video Surveillance with Augmented
    Virtual Environments
  • Demos/proposals/talks
  • NGA, NRO, ICT, Northrop Grumman , Lockheed
    Martin, HRL/DARPA, Olympus, Airborne1, Boeing