Video And Mesh Processing for 3D Cinematography - PowerPoint PPT Presentation


PPT – Video And Mesh Processing for 3D Cinematography PowerPoint presentation | free to view - id: 16c38d-ZmNhO


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Video And Mesh Processing for 3D Cinematography


3D cinematography, also sometimes called free-viewpoint video, is the process of ... Holographic theatre. Tele-presence. Remote actors, dancers, musicians ... – PowerPoint PPT presentation

Number of Views:257
Avg rating:3.0/5.0
Slides: 37
Provided by: ronf2


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Video And Mesh Processing for 3D Cinematography

Video And Mesh Processing for 3D Cinematography
  • Remi Ronfard
  • INRIA Rhone-Alpes
  • VAMP Associate Team
  • March 2006

Plan of presentation
  • What is 3D cinematography?
  • Goal statement for VAMP
  • People in VAMP
  • Work Items in VAMP
  • Joint activities in 2005
  • Joint activities in 2006
  • Future work and applications

What is 3D Cinematography?
  • 3D cinematography, also sometimes called
    free-viewpoint video, is the process of building
    3D models of dynamic scenes from multiple video
    camera inputs.
  • This is an area of active research with enormous
    potential applications in education,
    entertainment, interactive television,
    video-conferencing, etc.
  • It is one of three scenarios actively
    investigated by MPEG for future extensions of
    their audio and video coding standards to 3D

Goal statement
  • Our teams at INRIA and Brown University have
    separately developped methods and tools for 3D
  • Moving to 3D cinematography raises difficult
    technological and scientific challenges, where we
    need to collaborate actively.
  • In this new application, the processing pipeline
    must be fully automated, so that 3D
    reconstruction can be performed at video frame

Goal statement
  • The goal of the associate team is to contribute
    to 3D cinematography by joint work in two crucial
  • real-time video processing for large networks of
    camera systems, using distributed/parallel and
    embedded architectures
  • real-time mesh processing for reconstruction,
    deformation, texturing and view interpolation of
    3D objects from multiple video, using
    multi-resolution models amenable to efficient,
    scalable compression.

Real-time video processing for multiple cameras
  • The goal of the associate team is to develop and
    share software and hardware for the 3D
    cinematography systems built separately by INRIA
    and Brown University.
  • Parallelization/embedding of dedicated algorithms
    for solving problems of camera calibration,
    background subtraction, feature extraction,
    feature matching and so forth.
  • Extend algorithms to dynamic cameras at multiple
    resolutions and viewpoints.

Real-time mesh processing of 3D objects
  • Our goal is to obtain high quality approximations
    of the deforming surfaces of objects in a dynamic
    scene at any point in time, and to render them by
    projecting video textures on them.
  • Subdivision surfaces and their multi-resolution
    extensions are the right tools for achieving both
    high-quality renderings, which are necessary to
    convey the illusion of reality, and
    high-compression ratios, which are necessary to
    overcome the massive amounts of data involved in
    multiple-view video.
  • This necessitates substantial extensions of the
    current state-of-the art in both computer vision
    and graphics, since the projective constraints
    arising from multiple cameras are essentially

Team Composition
  • The associate team is composed of 5 senior
    researchers, 10 PhD students and 2 engineers.
  • Gabriel Taubin and 4 students at Brown (EE)
  • Peter Sibley, Daniel Crispell, Yong Zhao, Douglas
  • Chad Jenkins and 3 students at Brown (CS)
  • Matt Loper, German Gonzales, Dan Grollman
  • Rémi Ronfard and 2 students at INRIA (MOVI)
  • David Knossow, Daniel Weinland
  • Edmond Boyer and 1 student (MOVI)
  • Clement Menier
  • Frederic Devernay, Loic Lefort and Herve Mathieu

Previous work
  • Gabriel Taubin and Rémi Ronfard has a close
    collaboration at IBM TJ Watson Research Center in
    1992 and 2000 on efficient data structures and
    algorithms for manipulating multi-resolution
    curves and surfaces
  • Gabriel Taubin, Remi Ronfard Implicit simplical
    models for adaptive curve reconstruction. IEEE
    transactions on Pattern Analysis and Machine
    Intelligence, 3(18), pp. 321-325, 1996
  • Ioana Martin, Remi Ronfard, Fausto Bernardini
    Detail-Preserving Variational Surface Design with
    Multiresolution Constraints. International
    Conference on Shape Modeling and Applications,
    June 2004.

Team Complementarity
  • The MOVI team at INRIA brings expertise in camera
    calibration and synchronization multiple-view
    stereo and shape from silhouette real-time and
    distributed systems action recognition.
  • The two teams at Brown bring complementary
    expertise in embedded systems programming and
    image-based rendering, mesh processing, machine
    learning and behaviour-based robotics.

Initial work items
  • Distributed video capture, including camera
    synchronization and calibration
  • Multi-resolution object surfaces from silhouettes
    using subdivision surfaces, with applications on
    modeling deformable objects and interpolating new
  • Human motion tracking and action understanding

(No Transcript)
(No Transcript)
(No Transcript)
Blinky Distributed Camera Control
Camera Recording and Playback
Camera Calibration
3D models from multiple views
  • Visual Hulls and Shapes (Edmond Boyer)

From multiple cameras to 3D
(No Transcript)
Beyond visual hulls and towards multi-resolution
(work in progress)
(No Transcript)
(No Transcript)
(No Transcript)
(No Transcript)
Markerless articulated motion
Human Action Recognition
Human action recognition example
Recovering body skeleton from video (work in
Achievement in 2005
  • June/July 2005 - The initial associate team
    meetings were very successful and appealed to PhD
    students enormously on both sides.
  • Those meetings revealed that both labs are
    heading very rapidly towards very large camera
    arrays (100's at INRIA, 1000's at Brown).
  • Although the initial investment in terms of
    hardware and software developments are still
    high, our collaboration will save efforts and
  • INRIA recording and playback software has been
    deployed at Brown for initial experiments,
    calibration and background subtraction to follow

Exchanges in 2005

Scheduled exchanges in 2006
  • Chad Jenkins will visit INRIA in April
  • continue ongoing collaboration with Clement
    Menier on medial-axis reconstruction from
    multiple silhouettes
  • start a new collaboration with Daniel Weinland on
    motion modeling and action recognition.
  • In June, Gabriel Taubin and Remi Ronfard host an
    international workshop on 3D cinematography at
    CVPR in New York City.
  • More student exchanges throughout summer (thanks
    to NSF)
  • multi-resolution surface from silhouettes
  • multi-view stereo, deformable surfaces

Scheduled exchanges for 2006

Future work Cinematized Reality
  • Track and follow action
  • Tracking paths of multiple people
  • Tracking articulated motion
  • Tracking activities
  • Generate new views
  • View-dependent textures mapped to 3D model
  • Camera placement problem - How to choose the best
    views ?
  • Generate new movies in 2D or 3D

  • Digitizing Live Performances
  • Cultural Heritage and Education
  • Interactive Television, Games
  • Holographic theatre
  • Tele-presence
  • Remote actors, dancers, musicians
  • Theatre of operations in medecine and military
  • Producing content for 3D cinema (2010) and 3D
    television (2020)

(No Transcript)