JunHyeok Heo

1
Time Critical Rendering

• JunHyeok Heo
• VRGroup CSDept. KAIST

2
Outline

• Introduction
• Approaches
• Culling
• Hierarchy Structuring
• Visibility Preprocessing
• Discussion
• Dynamic Environments
• Ongoing Research

3
VR system
Introduction

• Basic Interactions
• Continuous viewpoint control
• Manipulation of objects in the environment

VW
4
VR System
Introduction

• Requirements
• Fast display update
• 10 frames/second
• Fast response to user
• 0.1 second delays
• Which requirements?
• Grab and put the fast moving object

5
The Time-Critical Philosophy
Introduction

• Fast, slightly inaccurate better than slow, very
  accurate
• what inaccuracies are acceptable?
• Time budgets are assigned to tasks
• Algorithms choose best compromises in order to
  meet the time budget

By Steve Bryson
6
Not real-time programming
Introduction

• Real-time programming
• provides guaranteed response
• in a fixed time
• in a fixed software environment
• Time-critical techniques
• choose appropriate levels of degradation
• in a flexible, unpredictable software environment

By Steve Bryson
7
Basic Time-Critical Design
Introduction

• User specifies an overall desired frame time
• frame time 1/frame rate
• Each task within that frame is assigned a time
  budget
• task times should total to desired frame time
• Each task determines now to do the best job
  within that time budget

By Steve Bryson
8
Graphics H/W
Approaches

• DRAW Pipeline
• Geometry and Mode Optimization

9
Solutions for TCR
Approaches

• Geometry based rendering
• Polygon flow minimization
• Image based rendering
• Image caching
• hybrid method

10
Polygon Flow Minimization
Approaches

• Minimize the total number of polygons sent to the
  graphics pipeline
• Remove perceptually unimportant parts of
  environment
• Culling, Level of Detail

11
Polygon Flow Minimization
Approaches

• Time Critical Rendering of VR systems

CULL
DRAW
LOD
CULL
LOD
DRAW
12
Assumptions of VR systems
Approaches

• 1. use z-buffer algorithm for hidden-surface
  removal
• 2. local illumination algorithm
• 3. has a programmable, general culling processor
• 4. has a programmable, general generation and
  management processor of level of detail

13
Objective
Culling
Remove perceptually unimportant polygons
local illumination
Remove invisible polygons
Z-buffer hidden surface elimination
Keep a superset of visible polygons
Conservative Visibility
14
State of the art
Culling

• The Culling process is also Time Critical
• In case of Static environment, Preprocessing
• Two approaches by preprocessing types
• Hierarchy Structuring
• Visibility Preprocessing

15
Algorithm Outline
Hierarchy Structuring

• 1. Make the scene hierarchy by preprocessing
• 2. Culling all subparts of hierarchy by culling
  the parent parts on the fly
• Point to Area Visibility Problem

16
Based on Objects
Hierarchy Structuring

• Hierarchy of bounding volume
• Previously used to accelerate ray tracing

17
Based on Objects
Hierarchy Structuring

• Tightness of bounding volume
• Bounding box Clark 76
• Deformed bounding box by affine matrix Rubin 80
• Slabs Kay 86

18
Based on Space
Hierarchy Structuring

• Adaptive subdivision of space
• BSP Tree
• Octree
• Garlick et al. 90
• QuadTree
• Terrain - Falby et al. 93 - NPSNet

19
Culling on the fly
Hierarchy Structuring

• Frustum Culling
• Removing only parts out of view frustum
• Occlusion Culling
• Removing the occluded parts, too.
• Image Space
• Hierarchical Z-Buffer Greene 93
• Hierarchical Polygon Tiling Green96
• Hierarchical Occlusion Maps Zhang 97
• Object Space
• Linear Critical Surfaces Coorg 96, 97
• Shadow Frusta Hudson 97

20
Hierarchical Z-buffer
Hierarchy Structuring
Greene 93

• Object Space - Octree
• Image Space - Hierarchical Z-buffer

21
Hierarchical Polygon Tiling
Hierarchy Structuring
Greene 96

• Object Space - Octree of BSP trees
• construct a BSP tree with polygons associated
  with each leaf node of Octree
• strict front-to-back traversing
• Image Space - Hierarchical Triage Coverage Mask

22
Hierarchical Occlusion Maps
Hierarchy Structuring
Zhang 97

• Object Space - any hierarchical structure
• Image Space - Hierarchical Occlusion Maps
• Discrimating Occluders between Occludees

23
Hierarchical Occlusion Maps
Hierarchy Structuring
Zhang 97

• Occluder Selection
• Criteria
• Size
• Redundancy(eg. Picture on the Wall)
• Rendering Complexity
• Preprocessing
• 1. partition with regular grids
• 2. calculation of visibility with item buffer at
  grid point
• 3. keep the visible polygons for each grid point
• Selection by Distance
• within the visible polygons for the nearest grid
  point

24
Linear Critical Surfaces
Hierarchy Structuring
Coorg 96, 97

• Discriminating Occluders between Occludees
• Linear Critical Surfaces
• Supporting Planes
• Separating Planes
• Occluder Selection
• Approximated Solid Angle

A
25
Linear Critical Surfaces
Hierarchy Structuring
Coorg 96, 97

• Visual Event and Critical Surface

EV(VE) event
Critical Surface by a EV event
EEE event
26
Shadow Frusta
Hierarchy Structuring
Hudson 97

• Discriminating Occluders between Occludees
• Occluder Selection
• Approximated Solid Angle
• Depth Complexity
• need preprocessing for some sampled viewpoints
• Coherence
• good occluders maintain as good ones for several
  frames

A
B
C
27
Algorithm Outline
Visibility Preprocessing

• 1. Partition the space into subspaces
• 2. Calculated all visible polygons for each
  partitioned subspaces
• 3. Find the subspace where the viewpoint
  currently stays
• 4. Render the visible polygons of this subspace
• Area to Area Visibility Problem

28
Perfect Partitioning
Visibility Preprocessing

• Global Visibility Problem
• Find exactly partitioned subspaces by sets of
  visible polygons
• The subspaces is partitioned by Critical Surfaces
• O(n9) subspaces possibly exist Plantinga 90

29
Rough Partitioning
Visibility Preprocessing

• First partitioning and then calculating the
  visibility
• partitioning
• model based partioning
• BSP tree Airley 90Teller 92
• O(n3) time, and O(n2) space Peterson 90
• regular partitioning - Grid Yagel 96
• adaptive partitioning - Quad, Octree

30
Rough Partitioning
Visibility Preprocessing

• Inexact Solutions for Visibility
• stochastic ray casting Airley 90
• underestimate, invalid culling

visible object
object marked invisible
31
Rough Partitioning
Visibility Preprocessing

• Portal Stabbing Teller 92
• check whether a sightline stabs through a portal
  sequence
• include all polygons of objects visible through
  portals
• superset of visible polygons (PVS) - conservative
  visibility

32
Rough Partitioning
Visibility Preprocessing
33
Rough Partitioning
Visibility Preprocessing

• complexity - f-the number of occluders, n-the
  length of an active portal sequence, e-the total
  number of edges

34
Rough Partitioning
Visibility Preprocessing

• dynamic evaluation Luebke 95
• check visibility through portals on the image
  space on the fly
• corridor test Yagel 96
• deal somewhat restricted environments - tunnel,
  cave
• check the obscuration of the corridor by
  connected solid cells
• control the tightness of the superset with grid
  resolution

35
Rough Partitioning
Visibility Preprocessing

• Exact Solutions for Visibility
• Exact Shadow Boundary by Area Light Source
• BackProjection Drettakis 94, Stewart 94
• a graph representing polygonal light similar to
  aspect graph
• O(n18) and O(n6logn) to process one light source
  respectively
• Visibility Skeleton Durand 97
• a graph representing visual events
• O(n4) space and O(n5) time to construct the data
  structure
• No time and space analysis to answer the
  visibility query between two polygons

36
Limitations
Discussion

• Hierarchy Structuring
• check the entire data structure everyframe.
• impossible to load the entire model into the main
  memory (e.g. city)
• need some special hardware facilities for the
  efficiency.
• Visibility Preprocessing
• portal based methods
• the number of possible portal sequences
  combinatorially increases by the
  number of portals
• inadequate for models with many holes such as
  outdoor scene (e.g. city)

37
Some Questions
Discussion

• Other meaningful filters to construct the image
  space hierarchy ?
• Z-pyramid min filter
• Coverage mask pyramid a discrete filter for 3
  values
• HOM mean filter
• More persuasive criteria for occluders ?
• More clever methods for selecting occluders ?
• preprocessing runtime

38
Some Questions
Discussion

• More general method for dealing with more general
  environments other than architectural indoor
  environments or caves ?
• How much is the conservative visibility easier
  than the exact visibility ?
• Object space precision is necessary ?
• the visibility precision always degrades to image
  space precision with z-buffer algorithm

39
Some Questions
Discussion

• It should be always conservative ?
• Called as Approximate Visibility Culling by Zhang
• how guarantee that the error caused by the loss
  of visible parts is under some error bound ?
• Is Visibility Preprocessing an approach worth
  continuing to research after the capacity of main
  memory become enormous and be able to hold the
  entire complex data?
• The only applications in which the hierarchy
  structuring can not be applied are those that
  have too complex data to be loaded into main
  memory as a whole.

40
Some Questions
Discussion

• Is it easy to extend all these methods to dynamic
  environments ?
• How to Integrate Culling Process and LoD Process
  ?
• Cull and LoD
• Use detail models in the culling process
• Apply LoD tech. to the survived objects
• LoD and Cull
• Apply LoD tech. to all objects and then Culling
• LoD Selection for Visibility Preprocessing

41
Dynamic Environments

• Hierarchy Structuring Sudarsky 96,97
• Adjusting the structure with dynamic objects
• Delete dynamic objects and Add those again

Temporal Bounding Volume
Least Common Ancestor
42
Dynamic Environments

• Types of moving objects
• Active objects moving within a bounded volume
• Active objects moving along a predefined path
• Active objects expected as moving along a
  predicted path
• Active objects moving unpredictably

43
Dynamic Environments
a
c
d
b
44
Dynamic Environments

• Dynamic objects moving along a path
• predict the position in n frames
• update the bounding volume with the new moving
  volume
• treat same as dynamic objects moving in a closed
  space when it stays in the new moving volume
• Dynamic objects moving unpredictably
• culling them each frame using the bounding volume
• hierarchy structuring algorithm can be still used
• cell to cell visibility preprocessing

45
Ongoing Research

• Visibility Preprocessing
• an image based solution for area to area
  conservative visibility problem
• the apparent motion of polygons on the image
  plane.
• Culling executed on the fly
• not valid but assure some error bound on the
  image
• temporally coherent visibility
• control the trade-off between performance gain
  and error

46
Apparent Motion
Ongoing Research

• Static objects while the viewpoint moves
• Static viewpoint while objects move
• Apparent motion equation Waxman 88

47
Apparent Motion
Ongoing Research

• Apparent Motion for the PVS computation
• initially positioned at the origin on the XYZ
  coordinate
• translation only
• normalize the XY plane with W and H (the a b
  space)
• a, bT, an normalized viewer motion vector on
  the a b space

48
Apparent Motion
Ongoing Research

• an normalized viewer motion vector
• a translation vector of a point on the image
  plane
• the swept area of a polygon on the image plane
  with the apparent motion

y
Y
0,1T
0,0T
1,1T
1,0T
x
-Z
Z
0,1T
0,0T
X
1,1T
1,0T
image plane
49
Algorithm Outline
Ongoing Research

• For each pixel on the image plane
• 1. Determine which polygons cover that pixel by
  the apparent motion
• 2. Determine the ab ranges of the normalized
  viewer motion vector during which each of these
  polygons projects onto that pixel
• 3. Perform hidden surface removal by eliminating
  subranges associated with occluded polygons
• Similar to the image-based motion blur algorithm
  Korein 83
• Inefficient when all vertices of a polygon have
  different Z values each other

50
Ghost Polygons
Ongoing Research

• Campbell defines the umbra shadow volume using
  maximum blocker extremal planes Campbell 91

an area S
a blocker
maximum blocker extremal plane
umbra
maximum blocker extremal plane
51
Ghost Polygons
Ongoing Research

• Definition 7 front ghost polygon

an area S
umbra
a polygon B
the front ghost polygon of B
52
Ghost Polygons
Ongoing Research

• Definition 8 back ghost polygon
• Time to calculate umbra ( equivalent to ghost
  polygon )
• O(nm) Cambell91 and O(nm) Yoo95

the back ghost polygon of B
an area S
umbra
a polygon B
53
Ghost Polygons
Ongoing Research

• Theorem 1
• Conservative Visibility

54
Algorithm Outline
Ongoing Research

• VisibleSet is Initially empty
• InputSet is the set of input polygons
• For each pixel on the image plane
• 1. Determine which back ghost polygons of input
  polygons cover that pixel by the apparent motion.
• 2. Determine the ab ranges of the normalized
  viewer motion vector during which each of these
  polygons projects onto that pixel.
• 3. Perform hidden surface removal by eliminating
  subranges associated with occluded polygons.
• 4. Determine which front ghost polygons of
  InputSet cover that pixel by the apparent motion.
• 5. Determine the ab ranges of the normalized
  viewer motion vector during which each of these
  polygons projects onto that pixel.
• 6. Check if each of these polygons is not
  occluded by back ghosts during its ab ranges, if
  so add this polygon to the VisibleSet and remove
  from the InputSet

55
Determine ab ranges
Ongoing Research

• A ghost polygon has the same Z value for all
  vertices
• An translation vector uniquely defines an
  normalized viewer motion vector of a ghost
  polygon on the image plane

-y
y
P3
P3
P
P1
P
P1
P2
P2
-x
O
O
x
56
Determine the visibility of ab polygons
Ongoing Research
57
Valid Area of Computed PVS for an area
Ongoing Research
fov
Valid Checked Area
An area
fov
fov
An area
58
Future Works
Ongoing Research

• Exploiting Object Space Coherence
• How to Partition for the performance acceleration
  ?

FindPVS(OctreeNode n) s For each visible
face f of n if IsVisiblePolygon(f) then
if n is a leaf node then For each
polygon p associated to n if
IsVisiblePolygon(p) then ss U p else
For each child node cn of n s
s U FindPVS(cn) return s return
s
59
Culling on the Fly
Ongoing Research

• Exploiting the temporal coherence.
• The key idea of the proposed method
• avoid calculating the culling function everyframe
• for the current view v1 and the predicted view v2
  in n frames
• not valid but guarantee the error bound in images
• provide a user with a mechanism of controlling
  the tradeoff between performance and error

60
Culling on the Fly
Ongoing Research
v
v
2
2
V
V
v
v
1
1
61
The approximated error bound
Ongoing Research

• Real error bound e
• The maximum error on images caused by missing
  some visible polygons at any viewpoint on the
  path
• Approximated error bound
• The number of pixels occupied by newly visible
  polygons at v2

62
Algorithm Outline
Ongoing Research

• If v1 deviate from the predicted path then
• repredict the path from v1 to v2
• excute culling on both ends v1 and v2
• estimate the approximated error
• if this is larger than the user specified value
  then
• make the predicted path empty
• return the culled polygons for v1
• else
• return the culled polygons for v1 and v2

63
Some Improvements
Ongoing Research

• Calculate the perfect culling function to get
  visible polygons for v1 and v2 through two
  consecutive frames
• Incrementally calculate the perfect culling
  function for the next view v2 in advance when
  the constant direction of velocity

64
Some Improvements
Ongoing Research

• Regulating the frame rate to the target frame
  rate f by selecting the suitable n (the
  prediction interval)
• Efficiently calculating the visible polygons and
  the approximated error bound by using the
  hardware z-buffer as the item buffer

65
Future Works
Ongoing Research

• More correct error bound
• More unconstrained viewer motion