Analysis of compressed depth and image streaming on unreliable networks

1
Analysis of compressed depth and image streaming
on unreliable networks

• Pietro Zanuttigh, Andrea Zanella, Guido M.
  Cortelazzo

2
The problem

• Remote browsing of 3D scenes over wireless
  channels
• 3D representations usually require huge amount of
  data
• Real time browsing
• Wireless links are typically unreliable
• Robustness to packet loss issues

3
3D Scenes

• 2 Basic type of data

Geometry can be represented by a set of depth maps
Texture
Geometry
4
View warping

• Depth information allows to associate each sample
  in each view to a point in the 3D space
• If the camera positions are known it is possible
  to compute the position of the point in another
  view

5
Why using depth maps?

• Both texture and geometry info are compressed by
  using JPEG2000 coding
• Same coding/decoding scheme for both texture
  geometry
• Gain in simplicity
• JPEG2000 is a wavelet coding scheme that yields
  multilayer representation
• Gain in modularity
• JPEG2000 is standardized
• Gain in interoperability

6
Motivations

• Today 3D system transmits info over TCP
  connections
• TCP guarantees reliable transport at the cost of
  unpredictable time delay
• Might impair navigation fluidity
• UDP does not introduce extra delay but may
  experience packet losses
• Might impair visual quality
• A possible tradeoff might consist in using UDP
  together with protection schemes
• Data packets have very different relevance
  Unequal Error Protection (UEP)
• A proper design of such schemes require good
  knowledge of the effect of packet losses on the
  reconstructed view

7
Aim of this work

• This work aims at shading light on these aspects,
  answering the following questions
• How performance degrades with increasing loss
  probability?
• Which packets are more important?
• Is it better to protect geometry or texture info?

8
Remote visualization of 3D scenes

• Client-Server remote visualization system
• Scene represented as a set of views and depth
  maps scalably compressed in JPEG2000
• Interactive browsing at client side
• JPIP transmission protocol

9
Architecture of the system

• The server holds the 3D scene description as a
  set of images and depth maps scalably compressed
  in JPEG2000
• It decides which elements of the compressed
  streams are the most suitable to be transmitted
• The client renders the required view exploiting
  the data received from the server

10
Simulation scenario (1)

• The server transmits 1 view (texture) and 1 depth
  map (geometry)
• Depth information is used to warp the view to
  novel viewpoints
• 2 Test models
• Goku (synthetic model)
• Soccer Player (reconstructed from real world)

11
Simulation Scenario (2)

• Target scenario wireless link, UDP protocol with
  no retransmissions
• Lossy channel
• Random packet loss (1, 5 and 10)
• Loss of a consecutive packet batch
• Comparison of the rendered views with and without
  packet loss

12
Lossless reconstruction example

• Original texture size 40 KByte
• JPIP frame size variable from 0 byte to 1
  Kbyte
• Black area and highest resolution info
  transmitted in frames with very small size

13
Loss of texture information (1/2)

• MSE due to the loss of texture packets vs packet
  loss rate angle between available and required
  view (soccer player)
• Distortion increases with the amount of lost
  packets (expected)
• Distortion independent of the selected viewpoint

14
Loss of texture information (2/2)
Plot shows MSE due to the loss of a batch of
texture packets as a function of the position of
the lost packets batch and of the angle between
the available and required view

• Dependent on lost packet position
• JPEG2000/JPIP transmit compressed data packets in
  order of relevance, losing earlier packets is
  worse
• Unequal Error Protection could be exploited

15
Loss of depth information (1/3)

• Causes samples in the rendered views to be
  misplaced
• Critical on edges
• Big impact on MSE

16
Loss of depth information (2/3)

• Distortion increases with the amount of lost
  packets and depends on the position of the lost
  packet (as in the texture case)

17
Loss of depth information (3/3)
0
30

• MSE due to packet loss increases with the angle
  between available and required view
• (key difference with texture)

18
Depth and texture comparison
Plot shows MSE due to the loss of depth and
texture packets as a function of the amount of
lost packets and of the angle between the
available and required view

• Depth information more important, but probably
  overestimated by MSE metric
• Depth impact depend on the angle
• For small angles texture depth errors have
  similar impact
• For larger angles depth become much more important

19
Conclusions

• Very different relevance of different packets
  (JPEG2000 transmits them in order of relevance)
• Depth loss impact depends on the viewpoint,
  texture one does not

20
Final considerations (2)
21
Final considerations (3)

• Adding redundancy might be detrimental in case
  packet losses are due to contention instead of
  wireless link errors
• Cross Layer Optimization (CLO) techniques shall
  be used on the wireless link to shield end-to-end
  applications from wireless unreliability
• Other transport protocols, such as Stream Control
  Transmission Protocol (SCTP), might be considered

22
Future work

• Analysis of more complex simulation scenarios
  with multiple views and depth maps
• Design of ad-hoc hybrid TCP-UDP (SCTP) protocols
• UEP techniques for 3D models

23

• Wed like to dedicate this work to
• Federico Maguolo
• He was supposed to join us on this project
• but his tragic death has prevent us from all the
  excellent ideas and contributions
• he would for sure have given to this work.