Trends and Standards for 3D Graphics for Handsets

1
Trends and Standards for 3D Graphics for Handsets

• Christophe Quarre
• Advanced System Technology, Shanghai Lab
• STMicroelectronics
• Christophe.quarre_at_st.com

State Key Lab of CADCG, Zhejiang University 1-3
March 2005
2
Outline

• Handhelds VS Desktops.... new industry
• Challenges for ISVs
• Standards for graphics mobile.... and 3D
  acceleration
• The key players... ISVs IHVs
• Conclusion

3
Mobile 3D industry

• Mobile phones are about to be transformed in
  personal compute devices
• Graphics capabilities will be a key ingredient
  just as on the PC
• Before we had 2D, but now we want 3D !
• The Mobile 3D industry is embryonic - but moving
  fast!
• We are where PC graphics were in 1996
• But evolving about three times faster!

4
Game market in Korea
Surging Video and mobile game market
USD million
ltFrom 2003 Korea Game White Papergt
5
Desktop vs. Handheld Systems
Desktop Handheld
CPU INTEL 1GHz up to gt3GHz ARM as de-facto standard 100MHz to 500MHz
FPU YES Today ARM9 (Nomadik 1 _at_ ST) ? NO Tomorrow ARM11(Nomadik 2 _at_ ST) ? YES
CACH, MEMORY, BANDWIDTH Very big, power hungry 10s of KB of eSRAM 1 MB of eDRAM is now feasible 100s of MB DDR-SDRAM Ultra low power design library
RESOLUTION SVGA gt1024 768 pixels QVGA (320240) VGA (640480)
GPU (e.g.) Nvidia GeForce 6800, ATI Radeon X800 IMG, Falanx, ATI, nVidia, Bitboys and others
3D API OpenGL1.5 / 2.0 DirectX Java3D OpenGL ES 1.0/1.1and future 2.0 D3DM JSR184 M3G
6
Challenges on Handheld Platforms

• Small battery, small CPU
• Management of Host CPU ? unload max work on
  Host CPU
• and hardware resources ? can save 90 of battery
  life !
• Limited system memory size and bandwidth
• Memory architecture will make the difference !!
• Cache memory buffers compression
• A wide range of Mobiles and gfx architectures,
  from Low-End ? High-End
• Means scalable and flexible solutions !!!

7
Challenges for Software Vendors 1/2
Observation

• If we consider 3 likely classes of handheld
  device
• Class A High-performance CPU, GPU with vertex
  processing FPU
• Class B High-performance CPU, GPU with vertex
  processing
• Class C CPU, rasterizer
• Classes B and C, low cost will likely ship in
  higher volume,
• therefore may offer more revenue opportunities
  for software vendors
• but yet platforms do not have floating-point
  support (FPU)
• Class A device may win out for better capacity
  and performance
• ? Software vendors must cover all the bases to
  guarantee success

8
Challenges for Software Vendors 2/2
Action

• How to develop or port rich and attractive
  applications with such a spread of computational
  capabilities?
• System Software must be scalable with Unified
  driver architecture to rapidly be adapted to
  different system architectures
• Middleware is a key component
• E.g. RenderWare (Criterion) optimized for Nokia
  N-gage

9
Standardization

• Different class of mobiles and Handsets,
  different capabilities
• Many Graphics accelerator Suppliers
• Graphics low power for mobile is a new industry
• ? Need for standardizations

10
An Important and leading Group

• Member-funded industry consortium
• Lead specification of open standard,
  multi-platform, royalty-free low level APIs for
  HW acceleration
• Leverage graphics technology by working closely
  with ISVs IHVs

2004Embedded 2D Vector Graphics Acceleration
2002Embedded 3D Graphics Acceleration
2000Streaming Media Control and Video
Acceleration
2004Embedded Media Primitive Acceleration
11
Khronos Members
Promoting Members
Contributing Members
12
OpenML Standard for Dynamic media authoring

• royalty-free, cross-platform programming
  environment
• Abstraction layer for capturing, transporting,
  processing, displaying, and synchronizing digital
  media
• audio/video streams
• 2D/3D graphics streams
• OpenGL extensions for accelerated video
  processing
• Lowest level of unified functionality, thinnest
  layer on top of HW

Application Application Application Application Application
Windows system OpenGL OpenGL Extension MLdc ML
Synchronisation Synchronisation Synchronisation Synchronisation Synchronisation
13
OpenVG 1/2 Standard for Vector Graphics
acceleration

• Vector Graphics
• Exemple

Original image SVG zoom Bitmap
zoom

• Why OpenVG ?
• 2D vector graphics is necessary for high quality
  GUIs or Font display on small display
• Need hardware acceleration for useful performance
• Existing vector graphics APIs are proprietary and
  developers must rewrite code for each platform
• Existing APIs usually designed for software
  rendering

14
OpenVG 2/2 Standard for Vector Graphics
acceleration

• OpenVG Royalty-free, cross-platform, low-level
  APIs for hardware acceleration for high quality
  2D vector graphics.
• Interface for vector graphics libraries such as
  Flash and SVG
• Same framework as OpenGLES
• combine with 3D APIs (Blending, lighting,
  texturing)
• OpenVG1.0 specs is currently under review by
  selected ISVs. Official release expected in
  March 2005 !

15
OpenMAX Standard for Media acceleration primitive

• The problem

RISC
DSP
PORTING PROBLEM!? slow down multimedia sw
development ? POORLY optimized
Multimedia APIs
Multimedia Accelerators
H/W accel
SMP

• OpenMAX royalty-free, cross-platform, common
  APIs for multimedia application
• standardizes access to media processing
  primitives used extensively in graphics, audio
  and image libraries and video codecs (eg. MPEG-4)
  
• Benefit Accelerate porting of multimedia SW
• Currently in development, expected release in End
  2005

16
OpenMAX Standard for Media acceleration primitive

• Accelerating a broad range of media types

Applications
Image librariesVideo CodecsSound Libraries
3DSmall footprint 3D for embedded systems
VideoDynamic Media Authoring
Vector 2D Low-level hardware acceleration API
Accelerated media primitives for cross-platform
acceleration of media libraries
Three media libraries called by applications. Can
be ported directly to silicon.
Media Engines CPUs, DSP, Hardware Accelerators
etc.
17
OpenGL ES Standard for Embedded 3D graphics
acceleration

• Khronos has created a small-footprint subset of
  OpenGL
• Created with the blessing and cooperation of the
  OpenGL ARB
• Full functionality for 3D games
• On a wide variety of platforms including fixed
  point processors

Feedback and Ratification
ARB
EmbeddedFocus
WorkstationFocus
Eliminate Redundancy
Small footprint e.g. 50KB software engines
Eliminate Workstation Functionality
18
OpenGL ES Roadmap

• Update OpenGL ES every year by default
• To expose rapidly developing handheld platforms
  capabilities
• BUT ONLY introduce features with proven demand
  from ISVs or IHVs
• Guarding against unnecessary pieces of HW
• Track and adapt developments in desktop OpenGL

Full high-level shading language capability to
harness the power of programmable hardware
Shader programmability for embedded devices
OpenGL 2.0
OpenGL ES 2.0
Widely available cross-platform 3D graphics API
OpenGL 1.5
Increasing emphasis on enabling emerging hardware
with video support and enhanced 3D pipeline
OpenGL ES 1.1
OpenGL 1.3
Enabling software AND hardware implementations
including small-footprint, low-end fixed point
platforms
OpenGL ES 1.0
Mid-03
Mid-04
Mid-05
19
Java The multiplatform language

• Game developers prefer Java, better for creating
  and managing tools and
  high level gfx libraries
• But still need low level access to the 3D HW
• Binding to OpenGL and OpenGLES
• JSR 231 OpenGL (Expert Group created in Oct
  2003)
• JSR 239 OpenGL ES (Expert Group created in
  March 2004)
• Wrappers around native OpenGL ES implementation
• Higher level, Scene Graph API
• JSR 184 Mobile 3D Graphics API for J2METM
  (Released in Nov 03)
• can be built on a native OpenGL ES implementation
• can be layered on JSR 239
• Reduce application size for OTA downloading

20
SW Standards layers for 3D Mobile graphics
C/C Applications
JavaApplications
Applications
High-levelGraphics Libraries
Scenegraph APIsM3G (JSR 184)
MiddlewareLibraries
GamesEngines
Low-level3D Graphics API
Hardware OpenGL ES Engines
Software OpenGL ES Engines
J2ME
Operating Systems
Hardware
Hardware
21
Graphics SW (Engine, Middleware) Providers

• Games developers
• Game Engine, Middleware, Dev. tools
• OpenGLES implementation and driver, Authoring
  tools...
• JSR184/239

Mascot Capsule
X-FORGE
22
Graphics HW acceleration Providers
PowerVR MBX / MBX Lite
Acceleon G30/G40
Mali50/100
IMAGEON 2300
GoForce 4000/3000
And of course !!
23
Mobile Multimedia chips Providers
ARM9 (200MHz), VFP, MBX
ARM11 (330MHz), TMS310 (220MHz), VFP, MBX
ARM11 (500MHz), MMDSP(ST), VFP, 3D Acc (ST)
SH-CPU (600MHz), SH-DSP, VFP, MBX
Hitachi semiconductor Mitsubishi semiconductor
24
Conclusion

• 3D graphics for mobile is a new industry,
• for a new market...
• To be competitive, SW systems need to be
  scalable, with
• unified low level standards,
• unified drivers...
• Middleware is the key for success,
• to provide efficient data conditioning...

• Thank you !