Advances in RealTime Rendering in 3D Graphics and Games

1
Regressions in RT Rendering LittleBigPlanet Post
Mortem Alex Evans Anton Kirczenow Media Molecule
Advances in Real-Time Rendering in 3D Graphics
and Games
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
2
Constraints Mash-ups
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
3
LBPs Constraints

• (Back in 2006)
• Blank slate
• no code, no tech, no existing franchise
• New, Unknown Strange platform
• but a single platform!
• User generated content was to be key
• no pre-computation
• worst case is the typical case untrained
  artists aka users
• Distinctive art style based on miniature world
• style binding content allow LBPs style to shine
  through any UGC
• My technical prejudices
• 1.5 programmers 36 months

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
4
Prejudices to force decisions

• We had little time to mess about, so the
  overriding goal was always whatever is easiest
  to code plays to our strengths
• no middleware
• (home-made turtles all the way down please)
• one code path for all rendering of all objects
• eg same path for solid, transparent, fx...
• all lighting per-pixel
• no per-vertex or (shudder) per-object
  optimizations
• playing to strength of team I hate vertices, and
  I dont have time to write those secondary paths
  anyway
• minimum number of knobs for the artists - but
  not too few
• relatively non-technical art team, busy with
  other tasks

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
5
Prejudices into Features

• lots of dynamic geometry cloth
• (we cant pre-compute anything anyway...)
• motion blur, DOF, flexible color correction
• used in unusual context
• unbounded numbers of dynamic lights
• emphasis on many layers of texture to achieve
  surface look
• refractive glass
• correct DOF / motion-blur behind transparent
  objects
• ability to arbitrarily cover the entire world in
  'stickers' with no loss in frame-rate
• fluid cloth to drive all FX
• we almost avoided writing a particle system.
  almost.

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
6
Unusual vertex pipe

• Avoid Vertex Shader slowness on RSX by using SPU
  instead
• Add some sexy features to take advantage of SPU
• Cloth
• Soft bodies
• Patch tesselation
• Point deformers

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
7
Anton shows all
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
8
Vertex pipeline
iterate spring/collision constraints
Verlet Integrate
Collision with world
Input verts
Find cluster matrix
Rigid Skinning
Input bones
X
X
Morph targets
Stitch seams
compute N, T
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
9
Vertex motion

• SPUs job is to create world-space polygon soup
  per frame
• P N are triple buffered, allowing simple Verlet
  physics
• Pnew 2P-drag(P-Pold)-PoldA
• drag is set by collision routine
• Meshes automatically turned into springs across
  grids of quads, with artist weighting in vertex
  colors
• http//www.gamasutra.com/features/20000327/lander_
  02.htm
• Vertex lerped back to skinned pos
• using artist defined weights

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
10
Collision Detection

• SPU does collision against
• extruded 2D convex shapes (world)
• swept ellipsoids (character)
• vertex transformed into old new ellipsoid
  position
• raycast sphere vs ellipse in stationary ellipse
  space

raycast
Pold
Pold
old
Pnew
new
transform into ellipse space
Pnew
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
11
Clusters

• Lovely mesh-less algorithm least-squares fit a
  new rigid matrix to a deformed cloud of vertices
• remove center of mass - that gives translational
  component
• matrix sum of dyadic of (P-COM) x (Prest-COMrest)
• multiply by inverse of pre-computed matrix in
  rest pose
• orthogonalize resulting best-fit matrix to give
  affine transform, and optionally preserve volume
  by normalizing
• blend with original rigid body matrices to make
  as squishy/rigid as you want
• more detail Meshless Deformations Based on Shape
  Matching - Muller et al, SIGGRAPH 05

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
12
flexibility of vertex pipeline

• we ended up using that pipeline not just for
  meshes, decorations, cloth
• but also SFX like confetti, dust, embers,...
• allowing us to almost entirely avoid an explicit
  particle system
• with one exception jet-pack smoke was a simple
  point-sprite based backend on the
  integrated/collided vertices...
• rendered 1/4 screen-res for fill-rate reasons
• each point sprite has a 2D gradient across it -
  gives remarkable sense of 3d-ness
• every pixel samples the suns shadowmap to give
  volumetric shadows in the smoke

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
13
On to pixels graph based?
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
14
fixed everything aint so bad?

• the graph editor was never used by artists
• they were happy to composite LOTS of texture
  layers at different scales
• and left me to hard-code a single uber-shader
  that handled everything else.
• that BRDF is entirely cooked up based on live
  shader editing and my eyeballs. approximately
• 2 strong rim-light with color set using artist
  controlled hemisphere based on (V.N)2 with bias
  scale
• the 2 rim lights are lerped between based on
  sunlight shadow
• diffuse a mixture of N.L and (N.L)2
• ambient a simple 2 color hemisphere
• specular simply (R.L)22

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
15
shadowing

• one sun light arbitrary numbers of sprite
  lights
• world has shallow depth, so we got away with very
  simple shadowmapping
• Initially TSM eg http//www.comp.nus.edu.sg/tan
  ts/tsm/tsm.pdf
• Finally, just orthogonal single 768x768
  shadowmap!
• very tightly tuned frustum of shadow casters
• soft shadows via variance shadow maps AO
  contact shadows
• just the vanilla form of VSM after Donnelly et
  al
• lots of light leaking
• used 16FP render targets with two copies of X,
  X2 in R/G and B/A
• second copy scaled by 32. recombined in shader -gt
  less boiling

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
16
ambient occlusion for contacts

• the characters feet were small felt floaty...
• so we added an AO light that analytically
  computed the occlusion of the character modeled
  as 5 spheres (2 feet, 2 legs, 1 groin)
• see Inigo Quilez awesome site for derivation
• http//iquilezles.org/www/articles/sphereao/sphere
  ao.htm

AO (N.D) (r2/D.D)
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
17
Spritelights take 1

• as presented at SIGGRAPH 06, I started by slicing
  the view frustum and rendering light volumes
  directly into those slices (as a volume texture)
• A sort of irradiance volume - but with no
  directional component. (0th order SH)
• Use grad of irradiance in direction of normal to
  approximate N.L type lighting.

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
18
Sprite Lights Take 1
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
19
Sprite Lights Take 1
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
20
Sprite lights take 2 Defer?

• Lay down a Z/N pre-pass at 2X MSAA

...then light as if it were grey plastic
(specular -gt alpha)

• ...and do sprite lights in the traditional
  deferred way
• BUT! no G-buffer, no material properties, no
  MRTs just Z, N in

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
21
Sprite lights take 2 Defer?

• then re-render scene forwards, sampling L
  into BRDF

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
22
sprite lights take 2 god rays

• now were doing sprite lights properly, we might
  as well add some features god rays!

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
23
god rays implementation

• completely 2d screenspace effect
• each light volume is rendered to an off-screen
  surface, one channel per light (in batches of 4)
• pixel shader integrates light scattering along
  eye ray to first solid object (z buffer distance)

• 3 pass smear shader then smears
• dark regions with MIN blending
• each pass doubles distance of smear
• similar to masas rthdribl lens-flare streak
  shaders
• http//www.daionet.gr.jp/masa/rthdribl/

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
24
god rays implementation

• completely 2d screenspace effect
• each light volume is rendered to an off-screen
  surface, one channel per light (in batches of 4)
• pixel shader integrates light scattering along
  eye ray to first solid object (z buffer distance)

• 3 pass smear shader then smears
• dark regions with MIN blending
• C sample(u,v)
• for (i0ilt5i)
• u(u-cu)kcu v(v-cv)kcv
• Cmin(C, sample(u,v) i/5.f)
• (where klt1 sets smear radius, cu,cv is the
  screenspace position of the lightsource)

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
25
2 layer transparency

• deferred shading with alpha. what?

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
26
2 Layer transparency

• Exploiting the fact that we have 2X MSAA hardware
• and we can cast the L buffer to be 2560x720p
  non MSAA
• deferred lighting done at full 2560x720p
  resolution
• allows us two independent Z values per final
  pixel
• PS3 you have to resolve your 2560x720p into
  1280x720 by hand. turn this into a virtue
• trade transparency for MSAA in different regions
  of the screen
• custom resolve shader is quite tricky uses alpha
  channel to control blend between the two samples
  - compositing either as A over B or (AB)/2

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
27
2 layer transparency DOF ...
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
28
2 is not a big number

• leads to odd effects when you get more than 2
  layers
• users actively are exploiting this to great
  effect!

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
29
2 layers Refraction

• Glass shader only WRITES to even pixels in X
  (alpha layer)
• it only READS from odd pixels in X (solid
  layer), sampling from its own screenspace pos P
  perturbed by the glass surface normal
• k is the index of refraction
• P.xykN.xy
• P.x-frac(P.x)0.5 lt--- MAGIC!
• Couttex2D(screen,P)
• code above relies on using UVs measured in
  pixels, not 0-1

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
30
careful MB/DOF calculation

• Motion-blur DOF were vital to the miniature
  look
• willing to spend lots of GPU cycles on this
• however it still runs at half res
• PASS 0 downsample 2x in Y, 4x in X
• blend 2 layers correctly according to degree to
  which each sample needs blurring. Alpha channel
  coarse z buffer

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
31
careful MB/DOF calculation

• PASS 1 2 7 tap classic DOF with variable
  circle of confusion, checking alpha channel for z
  conflicts
• samples distorted along direction of screenspace
  motion
• motion blur comes for free using same sample
  points as DOF

stationary pixel
fast moving pixel

• Final MSAA Resolve
• reads 1/2 res blurred image, and composites it
  over the full-res screen
• however it does this BEFORE blending the two
  alpha layers, and mixes the blurred image into
  each layer only according to that layers need.
• This conditional blend happens at full res, so no
  need for bilateral filter et al

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
32
first stab at motion blur

• we have a screenspace motion vector (x,y) for
  each pixel
• however, boundaries of objects dont look nice

• Initially, tried to render geometry extruded
  along direction of motion. however, fast rotating
  objects broke this

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
33
improving motion blur

• how to improve leading and trailing edges?
• insight use 2 frames of info - this frame (1)
  and old frame (0)
• blur C1 with V0, and C0 with V1 (twice as many
  taps)
• use Z-in-alpha to help composite samples together

Frame 0
Frame 1
these regions get frame 0 background blurred by
frame 1s velocity
while these regions get fr.1 background, with
fr.0 velocity
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
34
improving motion blur results
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
35
fluids how?

• Stam stable fluids, 2D water-grid PDE, or SPH?
• Anton will take them all, thanks very much!
• Stam-style stable fluids can be seen all over the
  place for smoke, fire dissolve fx
• Water Grid used for sea level
• Rigid body coupling to grid, generates...
• ... splash particles, combining SPH water-volume
  preserving forces with existing vertex pipeline
  (verlet, collisions etc)

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
36
water grid

• shallow water simulation - 256x128 on SPU
• fixed grid only contains visible play area, and
  is post-perspective distortion to give even
  tesselation
• The method is as in Claes Johansons MSc thesis,
  "Real Time Water Rendering Introducing the
  projected grid concept but tweaked
• to keep the grid evenly spaced in X when the
  camera is very near the surface and looking
  forward.
• V advects quantities with bicubic filtering
• linear not good enough - leads to viscous look
• method is similar to Chapter 11 of Fluid
  Simulation for Computer Graphics by Robert
  Bridson 2008

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
37
Water rendering

• The water surface normals are generated by
  bicubic filtering a texture of the height field,
  linear isnt good enough
• Refraction is just cheesy resampling of screen
  buffer with perturbed UVs by the surface normal
• additional z check to ensure refracted sample is
  actually below surface

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
38
Water rendering
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
39
water grid / rigid body coupling

• For rigid body coupling
• find rigid bodies intersecting the water surface,
  take their AABBs and compute the change of
  "submerged-ness" in the water.
• This is diffused and added to the water depth to
  create wakes and ripples.
• The method is as per Thurey et. al."Real Time
  Breaking Waves for Shallow Water Simulations"
  2007 ,
• but with Stam style stable diffusion so it
  doesn't blow up so much.
• We also add some of the velocities of the
  immersed objects to the water grid

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
40
Grid -gt SPH Splashes!

• We generate splashes wherever the water height
  changes rapidly
• emit particles where these both hold
• change in water height gt threshold
• gradient(water height) dot water velocity
  gt 0
• Splash particles (and bubbles) also add depth to
  the water grid when they collide with it to cause
  secondary ripples.
• splash particles are in an SPH simulation
• as per Muller et. al. "Particle based Fluid
  Simulation for Interactive Applications"
  Eurographics 2003
• with a novel collision detection scheme that is
  more SPU friendly
• more details in the course notes!

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
41
splat based splash rendering

• 2 pass rendering
• 1st pass sorts coarse quads front-to-back, with
  UNDER compositing of N and alpha
• this ensures isolated particles look sharp, while
  still blending adjacent particles smoothly
• each particle is elongated along its direction of
  motion
• particles near the water surface stretch to
  simulate joining
• 2nd pass outputs color
• uses stencil buffer to avoid overdraw
• uses similar shader as the water surface to
  provide consistency

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
42
underwater caustics!

• inspired by Pixars Finding Nemo DVD extras
• Advanced Animation and Rendering Techniques by
  Alan Watt Mark Watt, 1992, Ch 10.1

• intersect refracted grid of sun-rays with fixed
  ground-plane beneath the water surface

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
43
underwater caustics!

• inspired by Pixars Finding Nemo DVD extras
• Advanced Animation and Rendering Techniques by
  Alan Watt Mark Watt, 1992, Ch 10.1
• simply intersect refracted grid of sun-rays with
  fixed ground-plane beneath the water surface
• on SPU, for each vertex, compute change in area
  of surrounding refracted quads vs un-refracted
  quads
• output as a gouraud shaded mesh, rendered by GPU
• blend as MAX rather than , to keep dynamic range
  down

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
44
Refraction

• a 1024x512 caustic map renders in only 0.6ms!

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
45
underwater

• when rendering main scene, project caustic
  texture through world

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
46
stable fluids

• 3 layers of 256x128 2D fluid, used for smoke
  dissolve FX
• borrowing tricks from Mark Harris GPU
  implementation (GPU Gems)
• GPU re-samples screenspace z v buffers to
  initialize boundary conditions
• fluid only run on 3 world aligned slices, that
  scroll with the camera
• GPU copies from the backbuffer into the fluid
  buffer to cause objects to dissolve
• SPU outputs list of point sprites on a grid to
  avoid wasting fill rate where there is no fluid

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
47
breathe....

• ...and be thankful that we didnt go through all
  the other fun stuff, like virtual texturing for
  stickering, or procedural mesh generation on the
  SPU...

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
48
remind me what that was about?

• the key here is that many disparate techniques
  can be blended to create something unique looking
• its valid to treat dev-time as a primary
  constraint
• for LBP, we were able to trade accuracy
  generality for good enough approximations...
• the devil joy is in the details!
• the SPU freed us from worrying about casting
  every problem as a VS/PS stream processing DX9
  combination
• which ironically unlocked a lot of fun pre-GPU
  literature
• avoiding special case code paths kept the
  complexity down and the code coverage good...
• ...making it cheap to re-visit major decisions,
  like forward vs deferred, and re-implement
  features several times.

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
49
conclusion

• in other words,
• have fun mixing up and blending as many ideas as
  you think will help you achieve the visual look
  youre after
• (or something)

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
50
thankyou!
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
51
Bibliography 1/5

• Akenine-Moller T., Haines, E. Hoffman, N. (eds)
  - "Real Time Rendering 3rd Ed." 2008 A. K. Peters
  (pub) - http//www.realtimerendering.com
• Bridson, R. et. al. - "Curl-Noise for Procedural
  Fluid Flow" 2007 - in ACM Transactions on
  Graphics 2007
• Bridson, R. - "Fluid Simulation for Computer
  Graphics by Robert Bridson" 2008, Ch. 11 -
  http//www.cs.ubc.ca/rbridson/fluidbook
• Donnelly, W. Lauritzen A. - "Variance Shadow
  Maps" 2007 - http//www.punkuser.net/vsm/vsm_paper
  .pdf
• Engel, W. - "Light Pre-Pass Renderer Mark III"
  2009 - in SIGGRAPH 2009 Advances in Real-Time
  Rendering in 3D Graphics and Games course notes,
  2009

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
52
Bibliography 2/5

• Evans, A. - "Fast Approximations for Global
  Illumination on Dynamic Scenes" 2006 in SIGGRAPH
  2006 Advanced Real-Time Rendering in 3D Graphics
  and Games course notes - http//ati.amd.com/develo
  per/techpapers.html-acm2006
• Harris, M. - Fast Fluid simulation on the GPU
  2004 in GPU Gems
• Iwasaki, Kei et al - "An Efficient Method for
  Rendering Underwater Optical Effects Using
  Graphics Hardware" 2002 - in Computer Graphics
  Forum 2002 - http//nis-lab.is.s.u-tokyo.ac.jp/nis
  /cdrom/cgi/cgf.pdf
• Johanson C. - "Real Time Water Rendering
  Introducing the projected grid concept" 2004 - in
  MSc Thesis
• Lander, J. - "Devil in the Blue-Faceted Dress
  Real-Time Cloth Animation" 2000 - In Gamasutra
  Magazine Features, 2000 - www.gamasutra.com/featur
  es/20000327/lander_01.htm

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
53
Bibliography 3/5

• Martin, T. Tan, Tiow-Seng - "Anti-aliasing and
  Continuity with Trapezoidal Shadow Maps" 2004 -
  in proceedings of Eurographics Symposium on
  Rendering 2004 - http//www.comp.nus.edu.sg/tants
  /tsm/tsm.pdf
• Muller, M. et al - "Particle based Fluid
  Simulation for Interactive Applications" 2003 -
  in Proceedings of Eurographics 2003
• Muller, M. et al - "Meshless deformations based
  on shape matching" 2005 - in proceedings of
  SIGGRAPH 2005 - http//www.matthiasmueller.info/pu
  blications/MeshlessDeformations_SIG05.pdf
• O'Brien Hodgkins - "Dynamic Simulation of
  Splashing Fluids" 1995 - in Proc. Computer
  Animation 1995
• Quilez, I. - "Sphere Ambient Occlusion" 2007 -
  http//iquilezles.org/www/articles/sphereao/sphere
  ao.htm

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
54
Bibliography 4/5

• Sigg, C. Hadwidger M. - "Fast Third Order
  Texture Filtering" 2004 - in GPU Gems 2 -
  http//medvis.vrvis.at/fileadmin/publications/TR-V
  RVis-2004-053.pdf
• Stam, J. - "Stable Fluids" 1999 - In SIGGRAPH 99
  Conference Proceedings, Annual Conference Series,
  August 1999, 121-128 - http//www.dgp.toronto.edu/
  people/stam/reality/Research/pdf/ns.pdf
• Swoboda, M. et al, "Deferred Lighting and Post
  Processing on PLAYSTATION" 2009 - in GDC 2009
  Presentations, http//www.technology.scee.net/file
  s/presentations/gdc2009/DeferredLightingandPostPro
  cessingonPS3.ppt
• Thuerey, N. et. al. - "Real Time Breaking Waves
  for Shallow Water Simulations" 2007 -
  http//graphics.ethz.ch/thuereyn/ntoken3/Publicat
  ions.html

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
55
Bibliography 5/5

• Valient, M. et al, "Deferred Rendering in
  Killzone," 2007 - in Develop Conference,
  Brighton, July 2007. http//www.develop-conference
  .com/developconference/downloads/vwsection/Deferre
  d
• Watt, A. Watt, M. - "Advanced Animation and
  Rendering Techniques" 1992 - Ch 10.1

Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
56
the end
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)
57
really the end
Advances in Real-Time Rendering in 3D Graphics
and Games New Orleans, LA (August 2009)