Next Generation 4-D Distributed Modeling and Visualization of Battlefield

1
Next Generation 4-D Distributed Modeling and
Visualization of Battlefield
Avideh Zakhor UC Berkeley September 2003
2
Participants

• Avideh Zakhor, (UC Berkeley)
• Bill Ribarsky, (Georgia Tech)
• Ulrich Neumann (USC)
• Pramod Varshney (Syracuse)
• Suresh Lodha (UC Santa Cruz)

3
Battlefield Visualization

• Detailed, timely and accurate picture of the
  modern battlefield vital to military
• Many sources of info to build picture
• Archival data, roadmaps, GIS and databases
  static
• Sensor information from mobile agents at
  different times and location dynamic.
• Multiple modalities fusion
• How to make sense of all these without
  information overload?

4
Major Challenges Data

• Disparate/conflicting sources
• Large volumes.
• Inherently uncertain resulting models also
  uncertain.
• Need to be visualized on mobiles with limited
  capability.
• Time varying, time dependent and dynamic.

Make decisions and take actions
5
Mobile AR Visualization
Laser-------- Lidar-------- Radar------- Camera---
--- GPS--------- Maps-------- Gyroscope--
3D model construction with texture
Visualization Database
UNCERTAINTY
UNCERTAINTY
Model update
Decision Making
Mobiles with augmented reality sensors
6
Research Agenda

• Modeling
• Construction, fusion, integration, dynamic
• Registration
• Image to model, model to model
• Texture
• Dynamic, completion
• Visualization and rendering of complex models
• Uncertainty processing and visualization
• Tracking, Bayesian networks, expert mobile
• Distributed decision-making and reasoning with
  uncertain data

7
Visualization Pentagon
4D Modeling/ Update
Visualization and rendering
Tracking/ Registration
Decision Making under Uncertainty
Uncertainty Processing/ Visualization
8
Modeling

• Construct
• Static and dynamic
• Model fitting, and mesh based
• Fuse
• Airborne/ground
• Multiple data bases
• Multiple modalities

9
Fusing Aerial and Ground Based Models
Airborne Modeling
Ground Based Modeling

• Laser scans/images from plane

• Laser scans images from acquisition vehicle

3D Model of terrain and building tops
Highly detailed model of street scenery
building façades
Fusion
Complete 3D City Model
10
Airborne Model Generation
Re-sampling to regular grid
Scanning city from plane
unstructured point cloud
Problem
Digital Surface Map (DSM)
Jittery edges, bumpy roofs
11
Airborne Model Generation
Airborne DSM
DSM Triangulation
DSM Post-processing
Multiple Aerial Images
Image Registration (manual or automated)
Image Selection
Texture Mapping
City Model
12
Airborne Modeling Based on Parametric Fitting
USC campus area buildings
13
Registration and Texture

• 3D model registration at multiple scales
• Image/model registration for texture mapping
• Multi-image texture mapping
• Texture completion
• Dynamic texture mapping

14
Hierarchical Registration of 3D Models
Using image pairs of different resolutions
15
Hierarchical Registration of 3D Models
High and Low Resolution Models and Images
Registered -- consistently registered mesh --
scale-sensitive computation -- low registration
errors -- high scale ratios
16
Automatic image/model registration
Good Match
For comparison Bad Match
x 1533.3 m y 3968.1 m z 262.3 m focal length 18.2 mm yaw 35.57 deg pitch 54.71 deg roll 0.30 deg rating 1604
x 1533.3 m y 3968.1 m z 262.3 m focal length 18.2 mm yaw 35.57 deg pitch 53.71 deg roll 0.30 deg rating 767
17
Texture-Mapping Airborne Models with Multiple
Images
Images selected for individual areas
Different colors different images for texture
mapping
18
Texture Completion
before
after
19
Dynamic Texture Projection on LiDAR Data

• Enables Real Time Multi Source Data Fusion
• Requires accurate 3D model, sensor model,
• and texture/model registration

Sensor
Sensor
Image plane
View frustum
Aerial view of projected image texture (campus of
Purdue University)
20
Dynamic Texture Projection on LiDAR Data
Dynamic texture mapping An AVE image showing
three video projections around a campus building
complex
21
Dynamic Tracking and Modeling
Dynamic Modeling Tracked dynamic objects
(vehicle people) and their pseudo-models
visualized in 3D AVE world
22
Outdoor Video Tracking Using Multiple Cameras

• Goal To track moving objects using multiple
  cameras under changing weather conditions.
• Approach Automatic identification of informative
  cameras and measurement fusion.
• Key Steps
• Automatic camera selection based on appearance
  ratio (AR).
• Data assignment based on gating.
• Measurement fusion using a Kalman filter.

Trajectories based on fused measurements
System architecture
Images and blobs
23
Automatic Event Recognition

• Goal Given a sequence of video images construct
  a high-level event description
• Approach Low-level processing, categorization
  and interpretation of image sub-sequences
• Key Steps
• Filtering, detection, tracking, noise and shadow
  elimination, and feature extraction
• High-level analysis using HMM and Bayesian
  approaches
• Detection of an activity via statistical methods
• Control chart
• Decision tree learning algorithms

24
Rendering/Handling Complicated Models
View-Dependent LOD for large collections of
complicated models
Q
Q
Q
Q
Q
Linked Global Quadtrees
N Levels
Q
Bounding box
Viewpoint
Selected LOD
25
Mobile Situational Visualization

• An extension of situation awareness
• Mobile users with a variety of sensors and their
  own 3D databases
• Access to multiple servers
• Ability to mark and annotate positions of people
  and vehicles
• Placement of multiresolution models from MURI
  team members into environment

26
Expert mobile tracking probabilistic uncertainty
modeling

• Motion parameter estimates have probabilistic
  distributions and start with Gaussian
  distributions.
• Experts predict the position and speed and start
  with uniform weights.
• Experts weights updated based on their
  performance.
• Sampling algorithm extends to non-Gaussian
  distributions.
• Learning rate and memory parameters control the
  rate of learning

27
Image Query for Location Determination

• Goal To find the location based on images of the
  surrounding area
• Approach Matching of received images with the
  stored city model
• Key Steps
• Segmentation Based on Hough Transform
• Multi-level Matching Based on Texture Information
• Berkeley Scene

- Query Images
- Results
28
Transitions/Interactions with Government

• Nima
• NRO
• ARO
• ARL
• ICT
• Army Night Vision Lab
• NRL
• NASA
• EPA
• Lawrence Livermore Lab

29
Transitions/Interactions with Industry

• Lockheed Martin
• Hughes Research Lab
• Olympus
• Airborne1
• HJW
• Hewlett Packard
• TRW Northrop/Grumman
• Raytheon
• Andro Computing Solutions
• Sensis
• Critical Technologies
• Digicomp
• Alphatech
• Scientific Systems Company
• INRIA
• Sarnoff
• Panasonic

30
Dissemination of knowledge

• Special issue of CGA on 3D Reconstruction and
  Visualization of Large Scale Environments,
  Ribarsky Co-Editor
• Suya You and Ulrich Neumann, "Approaches to
  Large-Scale Urban Modeling
• Christian Frueh and Avideh Zakhor, "Constructing
  City Models by Merging Ground-Based and Airborne
  Views
• G. Foresti, C. Regazzoni and P.K. Varshney,
  Editors, Multisensor Surveillance Systems The
  Fusion Perspective, Kluwer Academic Press, 2003.

31
Participation in Planning Defense Initiatives

• Ribarsky Co-chair, Information Technology and
  Cyber-Security, Georgia Tech Homeland Security
  Initiative, 2003.
• Ribarsky Invited participant, Joint Advanced
  Warfighting Program Technology Exploration
  Workshop, Institute for Defense Analysis,
  Washington, DC, September 4-6, 2002. Workshop and
  working groups on future defense technology needs.

32
Tech Transfer, Transitions and Interactions

• VGIS a key part of
• the Georgia Tech Homeland Defense Workshop
• Sarnoff Raptor system, which is deployed to the
  Army and other military entities.
• Raptor system at Scott Air Force Base.
• In discussion with the Department of the Interior
  on use of mobile situational visualization
  capability.

33
Publications (1)

• C. Frueh and A. Zakhor, "Reconstructing 3D City
  Models by Merging Ground-Based and Airborne
  Views", to be presented at VLBV 20003, Madrid,
  Spain
• C. Frueh and A. Zakhor, "Automated Reconstruction
  of Building Facades for Virtual Walk-thrus",
  presented at SIGGRAPH Sketches and Applications,
  San Diego, 2003
• C. Frueh and A. Zakhor, "Constructing 3D City
  Models by Merging Ground-Based and Airborne
  Views", to appear in IEEE Computer Graphics and
  Applications, Special Issue Nov/Dec 2003.
• C. Frueh and A. Zakhor, "Constructing 3D City
  Models by Merging Ground-Based and Airborne
  Views", in IEEE Conference on Computer Vision and
  Pattern Recognition 2003, Madison, USA, June
  2003, p. II-562 - 69.
• C. Frueh, Automated Reconstruction of Urban
  Environments, Ph.D. Thesis, University of
  Karlsruhe, October 2002.

34
Publications (2)

• C. Frueh and A. Zakhor, "Data Processing
  Algorithms for Generating Textured 3D Building
  Façade Meshes From Laser Scans and Camera
  Images", in Proc. 3D Data Processing,
  Visualization and Transmission 2002, Padua,
  Italy, June 2002, p. 834 847
• J. Hu, S. You, U. Neumann, "Approaches to
  Large-Scale Urban Modeling," (accepted for
  publication) IEEE Computer Graphics
  Applications
• U. Neumann, S. You, J. Hu, I.O. Sebe, and B.
  Jiang, "Visualizing Reality in an Augmented
  Virtual Environment," (accepted for publication)
  PRESENCE Teleoperators and Virtual Environments
• D. Fidaleo and U. Neumann, "Analysis of
  Co-Articulation Regions for Performance Driven
  Facial Animation," (accepted for publication)
  Journal of Visualization and Computer Animation
• T.Y. Kim and U. Neumann, "Interactive
  Multiresolution Hair Modeling and Editing," ACM
  Transactions on Graphics and Computer Graphics,
  proceedings of ACM SIGGRAPH 2002, Vol 21, No. 3,
  pp. 620-629, San Antonio TX, July 2002.

35
Publications (3)

• I.O. Sebe, J. Hu, S. You, U. Neumann, "3D Video
  Surveillance with Augmented Virtual Environments"
  (accepted for publication) ACM SIG Multimedia -
  Workshop on Video Surveillance
• F. Bertails, T-Y. Kim, M-P. Cani, U. Neumann,
  "Adaptive Wisp Tree - A Multiresolution Control
  Structure for Simulating Dynamic Clustering in
  Hair Motion," Eurographics/SIGGRAPH Symposium on
  Computer Animation 2003 San Diego, July 2003
• SZ. Deng, J.P. Lewis, U. Neumann, "Practical Eye
  Movement using Texture Synthesis," ACM SIGGRAPH
  '03, Sketches and Applications, San Diego, July
  2003
• S. You, J. Hu, U. Neumann, and P. Fox, "Urban
  Site Modeling From LiDAR," Lecture Notes in
  Computer Science Series, Springer-Verlag, Vol.
  2669, ISSN 0302-9743, G. Goos, J. Hartmanis, and
  J.Van Leeuwen (Eds.) Proceedings of Second
  International Workshop on Computer Graphics and
  Geometric Modeling, Vol. 2668, pp. 579 - 588,
  Montreal, CANADA, May 2003.
• U. Neumann, S. You, J. Hu, B. Jiang, and J. W.
  Lee, "Augmented Virtual Environments (AVE)
  Dynamic Fusion of Imagery and 3D Models," IEEE
  Virtual Reality 2003, pp. 61-67, Los Angeles
  California, March 2003.

36
Publications (4)

• Enciso, John P. Lewis, U. Neumann, and J. Mah,
  "3D Tooth Shape from Radiographs using Thin-Plate
  Splines," "MMVR11 - NextMed Health Horizon", The
  11th Annual Medicine Meets Virtual Reality
  Conference, pp. 22-25, Newport Beach, California,
  January 2003.
• J. W. Lee, S. You, and U. Neumann, "Tracking with
  Omni-Directional Vision for Outdoor AR Systems,"
  IEEE ACM International Symposium on Mixed and
  Augmented Reality (ISMAR 2002), pp. 47-56,
  Darmstadt, Germany, October 2002.
• D. Fidaleo and U. Neumann, "CoArt
  Co-Articulation Region Analysis for Control of 2D
  Characters," IEEE Computer Animation, pp. 17-22,
  Geneva, Switzerland, July 2002.
• R. Enciso, A. Shaw, U. Neumann, J. Mah. "3D Head
  Anthropometric Analysis," SPIE Medical Imaging,
  (to appear) San Diego, CA, February 2003.
• R. Enciso, A. Memon, D. A. Fidaleo, U. Neumann,
  and J. Mah, "The Virtual Craniofacial Patient
  3D Jaw Modeling and Animation," "MMVR11 -
  NextMed Health Horizon", The 11th Annual
  Medicine Meets Virtual Reality Conference, pp.
  65-71, Newport Beach, California, January 2003.

37
Publications (5)

• David Krum, Olugbenga Omoteso, William Ribarsky,
  Thad Starner, and Larry Hodges. Evaluation of a
  Multimodal Interface for 3D Terrain
  Visualization. pp. 411-418 IEEE Visualization
  2002.
• Zachary Wartell, Eunjung Kang, Tony Wasilewski,
  William Ribarsky, and Nickolas Faust. Rendering
  Vector Data over Global, Multiresolution 3D
  Terrain. Eurographics-IEEE Visualization
  Symposium 2003, pp. 213-222.
• Zachary Wartell, William Ribarsky, and Nickolas
  Faust. Precision Markup Modeling and Display in a
  Global Geospatial Environment. To be published,
  SPIE 17th International Conference on
  Aerospace/Defense Sensing, Simulation, and
  Controls (2003).
• William Ribarsky. Virtual Geographic Information
  Systems. To be published. The Visualization
  Handbook, Charles Hansen and Christopher Johnson,
  editors (Academic Press, New York, 2003).
• Peter Wonka, Michael Wimmer, Francois Sillion,
  and William Ribarsky. Instant Architecture.
  Siggraph 2003, pp. 669-678 (2003).

38
Publications (6)

• Justin Jang, William Ribarsky, Christopher Shaw,
  and Peter Wonka. Appearance-Preserving
  View-Dependent Visualization. To be published,
  IEEE Visualization 2003.
• Nickolas Faust and William Ribarsky. Integration
  of GIS, Remote Sensing, and Visualization.
  Invited paper, to be published, Proc. Remote
  Sensing 2003 (Barcelona, 2003).
• William Ribarsky, editor (with Holly Rushmeier).
  3D Reconstruction and Visualization of Large
  Scale Environments. Special Issue of IEEE
  Computer Graphics Applications (2003).
• Zachary Wartell, William Ribarsky, and Nickolas
  Faust. Precision Markup Modeling and Display in a
  Global Geospatial Environment. To be published,
  SPIE 17th International Conference on
  Aerospace/Defense Sensing, Simulation, and
  Controls (2003).
• William Ribarsky. Virtual Geographic Information
  Systems. To be published. The Visualization
  Handbook, Charles Hansen and Christopher Johnson,
  editors (Academic Press, New York, 2003).
• Peter Wonka, Michael Wimmer, Francois Sillion,
  and William Ribarsky. Instant Architecture.
  Siggraph 2003, pp. 669-678 (2003).

39
Publications (7)

• Justin Jang, William Ribarsky, Christopher Shaw,
  and Peter Wonka. Appearance-Preserving
  View-Dependent Visualization. To be published,
  IEEE Visualization 2003.
• Suresh K. Lodha, Nikolai M. Faaland, Grant Wong,
  Amin Charaniya,Srikumar Ramalingam, and Arthur
  Keller, "Consistent Visualization and Querying of
  Geospatial Databases by a Location-Aware Mobile
  Agent", Proceedings of the Computer Graphics
  International Conference 2003, Tokyo, Japan, July
  2003.
• Srikumar Ramalingam and Suresh K. Lodha,
  Adaptive Enhancement of 3D Scenes using
  Hierarchical Registration of Texture-Mapped
  Models, To appear in Proceedings of 3DIM 2003,
  October 2003.
• Suresh K. Lodha, Krishna M. Roskin, and Jose C.
  Renteria, Hierarchical Topology Preserving
  Compression of Terrains", To appear in Visual
  Computer, 2003.

40
Publications (8)

• Amin Charaniya, Srikumar Ramalingam, Suresh
  Lodha, William Ribarsky, Nicholas Faust, Zach
  Wartell, and Tony Wasilewski, Real-Time
  Uncertainty Visualization of Mobile Objects
  within VGIS (Virtual Geographic Information
  System'', poster paper and interactive
  demonstration at the IEEE Visualization
  Conference, Boston , MA, October 2002.
• Srikumar Ramalingam, Nikolai Faaland, Amin
  Charaniya and Suresh Lodha, Visualization of
  Heterogeneous Geo-Spatial Intelligence in a
  Mobile Environment'', interactive demonstration
  at the IEEE Visualization Conference, Boston, MA,
  October 2002.
• Suresh K. Lodha, Nikolai M. Faaland, Amin P.
  Charaniya, Pramod Varshney,KKishan Mehrotra, and
  Chilukuri Mohan, K"Uncertainty Visualization of
  Probabilistic Particle Movement", Proceedings of
  The IASTED Conference on Computer Graphics and
  Imaging", August 2002, pages 226-232.
• Lilly Spirkovska and Suresh Lodha, Audio-Visual
  Situational Awareness for General Aviation
  Pilots'', to appear in the Proceedings of the
  SPIE Conference on Visualization and Data
  Analysis, January 2003, Vol. 5009.

41
Publications (9)

• R. Niu, P. Varshney, K. Mehrotra and C. Mohan,
  Sensor Staggering in Multi-Sensor Target
  Tracking Systems,'' Proceedings of the 2003 IEEE
  Radar Conference, Huntsville AL, May 2003. 
• R. Niu, P. Varshney, K. Mehrotra and C. Mohan,
  On Temporally Staggered Sensors in Multi-Sensor
  Target Tracking Systems,'' accepted to appear in
  IEEE Transactions on Aerospace and Electronic
  Systems, August 2002.
• J. Yang, C. Mohan, K. Mehrotra and P. Varshney,
  A Tool for Belief Updating over Time in
  Bayesian Networks,'' Proc. International Conf. On
  Tools for Artificial Intelligence, Nov. 2002,
  pp.284-289. 
• L. Snidaro, R. Niu, P. Varshney, and G.L.
  Foresti, Automatic Camera Selection and Fusion
  for Outdoor Surveillance under Changing Weather
  Conditions,'' Proc. of the 2003 IEEE
  International Conference on Advanced Video and
  Signal Based Surveillance, Miami FL, July 2003.

42
Publications (11)

• R. Niu, P. Varshney, K. Mehrotra and C. Mohan,
  "Temporal Fusion in Multi-Sensor Target Tracking
  Systems", Proceedings of the Fifth International
  Conference on Information Fusion, July 2002,
  Annapolis, Maryland.
•  
• Q. Cheng, P. Varshney, K. Mehrotra and C. Mohan,
  "Optimal Bandwidth Assignment for Distributed
  Sequential Detection", Proceedings of the Fifth
  International Conference on Information Fusion,
  July 2002, Annapolis, Maryland.
• Suresh K. Lodha, Nikolai M. Faaland, Amin P.
  Charaniya, Pramod Varshney, Kishan Mehrotra, and
  Chilukuri Mohan, "Uncertainty Visualization of
  Probabilistic Particle Movement", Proceedings of
  The IASTED Conference on Computer Graphics and
  Imaging", August 2002.
•  
• C. Regazzoni and P.K.Varshney, "Multisensor
  Surveillance Systems Based on Image and Video
  Data", Proc. of the IEEE Conf. on Image Proc.,
  Rochester, NY, Sept. 2002.

43
Publications (12)

• P.K. Varshney, "An Introduction to Distributed
  Detection Theory", book chapter in Multisensor
  Fusion, pp.163-182, Kluwer Academic Press, 2002.
• L. Osadciw, P.K.Varshney, and K. Veeramacheneni,
  "Optimum Fusion Rules for Multimodal Biometric
  Systems", book chapter in Multisensor
  Surveillance Systems The Fusion Perspective,
  Kluwer Academic Press, 2003.
• G. Foresti, C. Regazzoni and P.K.Varshney,
  Editors, Multisensor Surveillance Systems The
  Fusion Perspective, Kluwer Academic Press, 2003.
•  
• P.K.Varshney, Information Fusion, 4-hour Tutorial
  to be given at the IEEE Int. Conf. on Integration
  of Knowledge Intensive Multi-agent Systems, Sept
  2003.

44
Cross Collaboration
UCB USC G.T. SYR UCSC
Model const. fusion X x x x
Registration/texture x x x
Visuali-zation, rendering x x X
Uncertain. processing X x
Uncertain. Visualization. x x X
45
Outline of Talks

• Avideh Zakhor,
• "Overview
• Avideh Zakhor, U.C. Berkeley,
• "Fused 3D model construction of urban
  environments
• Ulrich Neuman, U.S.C.
• "Augmented Virtual Environments (AVE) for Dynamic
  Event Visualization
• Bill Ribarsky, Georgia Tech
• "Testbed for Mobile Augmented Battlefield
  Visualization"
• Suresh Lodha, U.C. Santa Cruz
• "Uncertainty Quantification and Visualization
  Registration of 3D Scenes and Expert Mobile
  Tracking
• Pramod Varshney, Syracuse,
• Distributed decision-making and reasoning with
  uncertain image and sensor data