Real-time Visualization of Massive Imagery and Volumetric Datasets

1
Real-time Visualization of Massive Imagery and
Volumetric Datasets

• Thesis Defense
• by
• Ian Roth

2
Overview

• Introduction
• Out-of-core software architecture
• Asynchronous data I/O engine
• Hierarchical caching
• Interactive visualization techniques
• Viewing modes for image generation
• Per-tile rendering methods
• User interface and navigation modes
• Performance and benchmarking
• Features and extensions
• Conclusions and future work

3
Kolam Software Package

• Developed at University of Missouri, Columbia
• Version 1.0 by Joshua Fraser (2000 04)
• Written in C, OpenGL and GLUT
• Version 2.0 by Ian Roth (2001 04)
• Written in C, OpenGL and Qt
• Additional code by Jared Hoberock (2001 02)
• Dataset I/O utilities
• Other contributors
• Dr. K. Palaniappan, Ian Scott, Dave Metts,
• Vidyasagar Chada, Mike Sullivan, Ryan Calhoun

4
Motivation

• Massive datasets
• Too large to fit entirely in memory
• Examples
• IKONOS/MODIS panchromatic/multispectral satellite
  imagery
• CT/MRI/PET scans, Visible Human Project
• Out-of-core approach
• Require segmentation (tiling) for efficient roam
  operations
• Require multiple resolutions (LODs) for efficient
  zoom operations

5
Definitions

• Image or Dataset
• Encapsulation of a pyramid file (disk cache)
• Ir(x) image at resolution r, pixel coordinate x
• Layer
• A dataset with associated viewing parameters
• Associated colormaps, heightmaps, etc.
• Stores x/y offset and scale for embedded datasets
• Colormap
• Maps dataset values to RGB color values
• Heightmap
• Maps image pixels to height values

6
Kolam Software Architecture

• Engine
• Based on the workpile design pattern
• Multiple worker threads process requests from
  application and perform data I/O operations
• Application
• Displays data and GUI components
• Determines which tiles are visible and at what
  resolution
• Posts request which are handled by engines
  worker threads
• Application Extensions
• Plugins dynamically linked libraries
• Provide additional GUI components, dataset I/O
  routines, image processing methods, image viewing
  and projection modes, etc.

7
Kolam Engine

• Asynchronously processes
• requests from application
• Proceedure
• Worker threads are initially idle
• For each frame, the application
• determines visible tiles and
• posts requests
• Each posted request awakens
• a worker thread
• Repeat forever for each awakened thread
• Worker thread attempts to pop request from queue
  if no requests are on the queue, then exit loop.
• Worker thread reserves a free slot in cache
• Worker thread performs tile I/O
• Application is notified of new data in cache
• Worker thread enters idle state

8
Kolam Engine Versions

• Version 1.0
• Each layer has its own resources
• Per layer cache space, threads, tile lookup
  table, requests
• Fine for non-embedded datasets
• Version 2.0
• Engine has global resources, shared by layers
• Engine cache space, threads, requests
• Per layer tile lookup table
• Allows active layers to seize resources from
  inactive layers

9
Hierarchical Caching

• Three levels
• Disk cache
• Stores tiles on disk
• Original source may be
• row-major order image file
• on disk or tiles streamed
• over network
• Main memory cache
• Stores fixed number of tiles in memory
• Texture cache
• Stores fixed number of tiles on graphics hardware
• Passed directly to real-time display component

10
Disk Cache

• Pyramid file format
• Additional resolutions appended to end of image
  file
• Requires additional space (no more than 2
  original)
• 1D image takes 2 space of original
• 2D image takes 4/3 space of original
• 3D image takes 8/7 space of original
• Wavelet encoded file format
• Additional resolutions embedded in image
• Requires no additional space
• Complex to encode/decode, especially in n-D
• Examples
• JPEG 2000
• MrSID (LizardTech)

11
Main Memory Cache

• Asynchronous I/O
• Multiple read/write threads
• Allows display thread to continue working while
  data is loading
• Spreads out I/O lag time over shorter intervals
• Caching allocates/deallocates memory for tile
  data
• Cache methods available to application developers
• checkout tile, checkin tile, add request, query
  tile/request, invalidate tile/request
• Variations of the checkout method
• basic checkout, checkout first available, wait
  for tile
• Paging swaps old data for new
• Temporal strategies view projection independent
• LRU priority queue O( of layers)
• Spatial strategies view projection dependent
• Brute force distance calculation O( of cached
  tiles)
• Modular ROI array O(1)
• Graph traversal algorithms O( of cached tiles)

12
Texture Cache

• Similar to main memory LRU cache
• Replace allocator object
• Synchronous I/O
• Adequate for single data display window
• Can be implemented asynchronously
• Useful for multiple displays
• One-to-one correspondence between thread and
  context
• Each display is essentially synchronous
• Frame rate not bounded by longest display time
• Implemented by OpenRM Scene Graph

13
Interactive Visualization Components

• Viewing modes
• Orthogonal projection
• Oblique projection (separate source tree)
• Spherical projection
• Arbitrary geometry projection (future work)
• Volumetric visualization (separate source tree)
• Tile renderers
• Raster renderer
• Texture renderer
• Terrain renderer
• Ray casting renderer (separate source tree)
• Navigation modes
• Roam, zoom, and tilt
• Fly mode (separate source tree)
• Embedded dataset positioning
• Image crop (future work)
• Geometry and terrain manipulation (future work)

14
Visualization Running Time

• Measured by number of tiles processed per frame
• Definitions
• n(I) total number of tiles in image I
• m(I, t) number of visible tiles in image I at
  time t
• Processing O(n) tiles per frame is too much
• Processing W(m) tiles per frame is the goal
• Analytic solution
• O(m) single resolution solution
• O(m lg n) multiple resolution solution
• Not suitable for explicitly defined projections,
  i.e. triangular meshes
• View projection dependent, often difficult to
  implement
• Iterative solution
• O(m lg m lg n) single/multiple resolution
  solution
• Relies on quadtree/octree/scenegraph traversal to
  
• iteratively refine solution from coarse to fine
  resolutions
• View projection independent, relatively east to
  implement

15
Orthogonal Projection Analytic Solution

• Visible rectangular region width/height
• Visible rectangular region x/y offset
• Resulting region R is in tile coordinates at
  scale r

• t 2D translation/position vector
• in original resolution pixel coords
• z pixel zoom factor
• wT, hT tile width/height
• wV, hV viewport width/height
• Global image resolution
• Scaling matrix (to tile coordinates)

16
Oblique ProjectionAnalytic Solution

• Single resolution solution
• Intersect view frustum planes with image plane
• Result is a polygon in the image plane
• Clip polygon to image boundaries
• Run polygon fill algorithm to find tiles in
  polygon at a fixed resolution
• Flood fill algorithm (recommended)
• First use Bresenham to discretize polygon to tile
  coords
• Scanline fill algorithm
• Multiple resolution solution
• First subdivide frustum into regions
• Calculate distance from eye p when z 1
• q FovY/2, r resolution, d distance from
  eye
• Assume pixel aspect ratio 1
• d and p both measured in pixels
• Perform single resolution solution

17
Oblique ProjectionIterative Solution

• Quadtree traversal
• Determine if tile is visible
• View-frustum culling test
• Performed in either R3 or canonical view volume
  (CVV) space
• Test 4 tile vertices w/ 6 frustum planes (24
  tests)
• Test 4 tile edges w/ 6 frustum planes (24 tests)
• Test 12 frustum edges w/ tile plane (12 tests)
• Maximum of 60 tests per tile
• Determine tile resolution
• Project to viewing plane and calculate
  quadrilateral area
• Caclulate resolution at sample points

18
Spherical ProjectionAnalytic Solution

• Convert Euler coords to spherical
• u 0, 2p, v -p/2, p/2
• Intersect view frustum planes with sphere
• Result will be up to 6 circles in 3D space
• n unit normal of plane P, p any point on P
• cc/s circle/sphere center, rc/s
  circle/sphere radius
• Find distance d from cs to plane along n
• Project circles of intersection onto image plane
• Parametric equations for circle and sphere
• t 0, 2p, x vertex in R3
• n unit normal of circle plane, p up vector
• a n p, b n a
• Solve for u and v
• For each circle
• Pick sample t values
• u(t) and v(t) are points in image plane
• Compute polygon intersections
• Tesselate non-convex polygons

19
Spherical ProjectionIterative Solution

• Determine visible tiles
• Tiles no longer planar, can be composed of many
  polygons
• Use bounding spheres to approximate curved tile
  shapes
• Similar to vertex/frustum intersection test
• Hidden surface removal
• Back-face culling methods dont work on curved
  surfaces
• Additional clipping plane to divide sphere into
  front- and back-facing regions
• Orthogonal to vector v
• Need to determine distance from eye
• Determine tile resolution
• Projecting and calculating area is difficult for
  curved tiles
• Overlapping regions not accounted for
• Caclulate resolution at sample points

20
Arbitrary Geometry ProjectionIterative Solution

• Image projected onto arbitrary triangular mesh
• Similar to spherical projection
• Requires mapping from mesh vertices to u/v coords
• Determine visible tiles
• Use hierarchy of bounding spheres
• Hidden surface removal
• ID image a lookup table of visible triangles
• Determine tile resolution
• Caclulate resolution at sample points

21
Volumetric VisualizationIterative Solution

• Octree traversal
• Determine if tile (cube) is visible
• View-frustum culling test
• Performed in either R3 or canonical view volume
  (CVV) space
• Test 8 tile vertices w/ 6 frustum planes (48
  tests)
• Test 12 tile edges w/ 6 frustum planes (72 tests)
• Test 8 frustum vertices w/ 6 tile planes (48
  tests)
• Test 12 frustum edges w/ 6 tile planes (72 tests)
• Maximum of 240 tests per tile
• Using bounding spheres only 6 tests required, but
  less accurate
• Optimization use spheres during octree
  traversal, use cubes on leaf nodes
• Determine tile resolution
• Project to viewing plane and calculate convex
  hull (polygon) area
• Caclulate resolution at sample points

22
Raster Renderer

• Renders images directly to framebuffer
• Advantages
• Simple to implement
• No additional texture/VRAM cache required
• No additional overhead from texture load time
• Tile data changes are applied instantly (on next
  display update)
• No need to flush texture cache
• Disadvantages
• Can only be used in orthogonal viewing mode
• Not all affine transformations supported
• Rotations and shearing
• Only 2D transformations
• Can be slow on some graphics hardware

23
Texture Renderer

• Stores tile data in texture/VRAM memory on
  graphics hardware
• Advantages
• Often faster than rendering directly to
  framebuffer
• Allows any affine transformation to be applied to
  a tile
• Allows hardware-accellerated interpolation to be
  applied when rendering
• Can be used by all viewing modes
• Volumetric viewing mode requires 3D texture
  support, otherwise must use ray casting renderer
• Disadvantages
• Wasted space when compositing color channels
• Luminance component always stored in red channel
  in OpenGL
• Must use RGB color textures for color masking to
  be effective
• Need to flush texture cache when tile data
  changes or global filters are applied
• Texture cache
• Similar to main memory LRU cache
• Synchronous rather than asynchronous
• Can reuse some code by replacing the allocator
  object
• Texture borders
• Use OpenGL GL_CLAMP_TO_EDGE texture parameter
• Widely supported, produces artifacts when z