Extensible 3D X3D Graphics Requirements for Video on the Web

1
Extensible 3D (X3D) Graphics Requirements for
Video on the Web

• Don Brutzman
• W3C Workshop, Video on the Web
• 12 December 2007
• Web3D Consortium www.web3D.org
• Naval Postgraduate School
• Monterey California USA

2
Topics

• X3D Overview
• Functional Requirements
• Video in X3D Scenes
• Converting 3D Models to Video
• Recommendations

3
X3D Overview
4
www.web3D.org

• The Web3D Consortium is a public-private
  partnership of industry, agencies, universities
  and individuals working on open standards for
  real-time 3D communication on the Web.
• Web3D develops, implements, evaluates and writes
  the Extensible 3D (X3D) Graphics specification.

5
Todays www.web3D.org page
6
Community rules

• Thanks to steady innovation by Web3D members and
  users, new features continue to evolve and grow
  into X3D capabilities
• Working groups identify requirements and write
  implementable specifications
• Individuals can directly extend specification
  capabilities without formal permission

7
ISO

• Formal annual review and ratification by the
  International Standards Organization (ISO) have
  made X3D an approved standard for real-world use,
  both on and off the Web
• Experts from 12-15 nations review our specs
• Immediate adoption by other governing bodies
  helps to increase deployment

8
W3C

• Further collaboration by Web3D Consortium with
  the World Wide Web Consortium (W3C) has made X3D
  a "first-class citizen" on the Web, providing
  excellent (and growing) interoperability with
  other XML standards
• Current joint work includes efforts on Efficient
  XML Interchange (EXI) for compression and
  processing improvements.

9
Intellectual Property Rights (IPR)

• Web3D and W3C have similar policies
• Any known patented technology must be declared by
  members prior to consideration by working groups
• Any patented technology contributions must be
  licensed on a royalty-free (RF) basis for
  inclusion in an openly used Web standard
  http//www.web3d.org/membership
• Caveat any legal problem can be solved, but
  only in advance

10
Open Source

• Open for any use, without license fees
• Free freedom to innovate
• Not necessarily free cost (more like free
  puppy)
• Common shared example implementation(s)
• Not a reference implementation the
  specification/standard hopefully provides that
  but required nevertheless
• Open source implementations are necessary to
  break logjams when company participants cant
  resolve technical issues
• Also provides self-sustaining business model for
  continued activity, improvement

11
IPR summary

• Royalties incompatible with Web architecture
• Open standards open source are both needed
• Open source by itself is necessary but
  insufficient
• Proprietary implementations are great too
• Standards organizations and IPR agreements
  provide a stable playing field for long term

12
X3D Profiles for Extensibility

• Different levels of content complexity
• Browsers can support increasing levels of
  capability
• Authors can use the proper palette for intended
  delivery

MovieTexture node is Immersive Profile
13
Example syntax

• ltShapegt
• ltBox/gt
• ltAppearancegt
• ltMovieTexture/gt
• ltTextureTransform/gt
• lt/Appearancegt
• lt/Shapegt

14
Family of X3D specifications

• Abstract specification describes how it works
• Equivalent encodings
• XML .x3d, ClassicVRML .x3dv, Binary .x3db
• Scene Access Interface (SAI)
• Consistent programming in EcmaScript, Java

15
X3D Specifications honeycomb diagram
16
The key challenge is scalability

• Because the only information systems capable of
  scalably growing to match global scope are the
  Internet and the World Wide Web, X3D deliberately
  follows the architectural principles of World
  Wide Web.
• Architecture of the World Wide Web, Volume One
  http//www.w3.org/TR/webarch

17
Functional Requirements

• Video within X3D Scenes
• 3D Geometry from Video

18
Linkability

• Linkability
• Video content needs to be referencable by URI
• X3D approach allows ordered list of equivalent
  URI addresses for reliable portability, access
• Bookmark additions are desirable for video,
  similar to using Viewpoints as X3D bookmarks
• Further annotation of location or time as part of
  URI is under consideration

19
Planet 9 Raygun demos
raygun.planet9.com
20
Playability

• Internal playability
• Video is often used as a user-facing billboard
  within the X3D scene
• Can be special display or match a particular view
• Mixed rendering is challenging but feasible
• Performance requirements are demanding
• External playability
• Anchor links or scripts may launch external video
  displays for multimedia experience

21
Interoperability

• Small number of required formats
• PNG JPEG GIF images, WAV audio, MPEG-2 video
• Motivated by performance and royalty-free (RF)
• Alternate formats are allowed but might not be
  supported, url list can alleviate this problem
• X3D browsers not (yet?) plugging in video codecs,
  but plugin interoperability continues to improve
• Adoption of imagery, audio and video formats
  supported by W3C is shared goal for Web3D

22
Planet 9 RayGun scene 1
Webcam stream as MovieTexture Billboard
23
Animation

• Need synchronization of video playback with
  3D-centric event-based behavior model
• Necessary for consistent playability
• Connect simple X3D animation interpolators,
  sequencers
• Unlocks both Ecmascript and Java, either inside
  X3D Script node or externally in HTML page
• Future work assignment
• investigate possible alignment of X3D event model
  with W3Cs Synchronized Multimedia Integration
  Language (SMIL)

24
Metadata overview

• X3D included document metadata (similar to html
  meta tags) and typed Metadata nodes
• Also Humanoid Animation (H-Anim) joint naming
• Some metadata conventions are emerging
• SAVAGE Modeling and Analysis Language (SMAL)
• No metadata vocabularies yet formalized by Web3D
• Wont reinvent other unencumbered standards

25
Metadata requirements

• Video metadata of interest
• duration, loopability, whether file or streamed
• time, geographic location for precise positioning
  in virtual environment
• Camera direction (and perhaps settings)
• Ability to add other metadata extensions
• Metadata needs to be inspectable by browser
• Use cases available for video embedded within, or
  external to, video content stream

26
Construction of 3D geometry

• Current research demonstrating ability to
  automatically generate 3D models from carefully
  registered/recorded video streams
• Variety of techniques are being developed
• Applications include city modeling, tracking
  human pedestrians, classifying produced objects,
  querying associated databases, etc.

27
Planet 9 RayGun scene 2
Numerous efforts generate matching 3D geometry
using video sources
28
Security

• Scene authors often want to encrypt or digitally
  sign 3D content
• Content is king, but creation is hard
• W3C recommendations for XML Security appear
  appropriate encryption, signature, PKI
• Security needs to be compatible with digital
  rights management (DRM) scheme
• Stream security needed, but not yet addressed

29
Digital rights management

• X3Ds XML and Compressed Binary encodings allow
  use of W3Cs Security recommendations
• XML Encryption
• XML Digital Signature (for authentication)
• Public key infrastructure
• DRM appears to be feasible, but not used yet
• More uses exist than Hollywood-commercial
• See Suns DReaM project http//www.openmediacommon
  s.org

30
Streamability

• Numerous historic examples of streamed video in
  3D environments
• Streamed video in X3D browsers is feasible but
  infrequent
• Likely due to performance and format permissions
• Real Time Streaming Protocol (RTSP) might be
  mapped to distribute X3D event model among
  multiple players
• Exemplar is Distributed Interactive Simulation
  (DIS) protocol
• Clear shared standards strategy might motivate
  Web3D member support for necessary next-step
  efforts

31
Planet 9 RayGun scene 3
Browsers have ability to stream video within
scenes
32
Synthesis

• Merging recorded or live video with 3D geometry
• Motion tracking for human avatars
• View from a window
• Likely future capability of broad interest

33
Transparency

• Goal superposition of selected video pixels of
  interest without box artifacts
• Example video talking head or human avatar
  without confounding background
• Ability to encode some pixels as transparent
• Red Green Blue Alpha (RGBA) pixel encoding is not
  practical for video streams
• Designating certain color as transparent is
  feasible, e.g. green screen

34
Export

• Recording of animated activity in immersive
  environment, as seen from one or more animated
  viewpoints
• Practice becoming more commonplace, (e.g.
  video-game players save victorious sessions)
• Different from playing back events within 3D
• Hollywood cliché What I really want is to
  direct now becoming feasible
• Similar requirements likely (e.g. video metadata)

35
Recommendations
36
Recommendations

• X3D graphics requirements for Web video are
  technically and politically feasible
• Interactive 3D performance is biggest challenge
• Plethora of oddball non-RF codecs unsustainable
• Royalty free model for video content thus
  essential
• Consider X3D technical requirements for video
  when building the foundation for emerging
  multimedia Web

37
Contact

• Don Brutzman
• brutzman_at_nps.edu
• http//web.nps.navy.mil/brutzman
• Code USW/Br, Naval Postgraduate School
• Monterey California 93943-5000 USA
• 1.831.656.2149 voice
• 1.831.656.7599 fax