Synchronization in Multimedia Data Retrieval

1
Synchronization inMultimedia DataRetrieval

• INTERNATIONAL JOURNAL OF NETWORK MANAGEMENT, VOL.
  7, 3362 (1997)
• SpeakerBei-Huang Chang

2
Outline

• Introduction
• Architecture of Distributed Multimedia
  Information Systems
• Multimedia Synchronization Scheme
• Buffer Control System
• Simulation
• Conclusion

3
Abstract

• Multiple medium streams synchronize in real time
  for multimedia applications based on broadband
  high-speed Networks
• Present synchronization scheme for DMIS without a
  global clock
• Whats Distributed multimedia information system?
• Integration audio,video,text or image originating
  from different servers(one server belong to one
  media-type)
• Interconnected by high-speed network
• Broadband network

4
Introduction(1)

• Eliminate all delays and variation incurred
  between media stream transmission and
  presentation.
• Packetization delay
• Network access dealy
• Transmission delay
• Protocol processing delay
• Presentation delay
• ProblemIts complicated when media data is from
  different source in the same presentation

5
Introduction(2)

• Clocks drift problem-buffer control
• Two type of media data
• Continuous time-dependent
• real-servicelive video
• need fine-grain synchronization(lip-sync)
• Discretetime-independent
• Text or graphic
• Need superior reliability
• Based above two types of media data
• Inter-media synchronization
• Intra-media synchronization

6
Introduction(3)

• Several synchronization requirements to solve the
  media synchronization problem
• some tools to express complex temporal
  relationships
• Real-time communication protocols and operating
  system (provide scheduling)
• Efficient buffer management

7
Architecture of Distributed Multimedia
Information Systems

• Quality of service for media applications
• A set of parameters that characterize
  communication services(ex.network Qos parameters
  like bandwidth,delay,jitter)
• According to Qos requirement for various
  media,network can reserve rosource
• Traditional communication are designed for data
  transmission,not suitable for real-time
  requirement

8
Multimedia Synchronization Scheme

• Concept Model Description of Media
  Synchronization
• Playout Schedule and Synchronization
• Buffer Configuration at the Client Side
• Determination of Control Time in Stored Data
  Retrieval
• Synchronization in DMIS

9
Concept Model Description of Media
Synchronization

• To satisfy temporal precedence relationships
• SIU(Synchronization information unit)
• Divide each media object into a sequence of
  subjects with its own sync interval
• Transmission of an object consists of a stream of
  SIU
• SIUs sync interval number as packet header
  information
• Ex.lip-sync are divided by the same duration

10
Concept Model Description of Media
Synchronization

• OCPN(object composition Petri net)
• Defined by the tupleT,P,A,M,D,R

11
Concept Model Description of Media
Synchronization

• OCPN(object composition Petri net)
• Inter-media describe a temporal presentation
  scenario

12
Playout Schedule and Synchronization

• Playout schedule means that play each medium data
  unit is specified so that play rate in terms of
  data unit per second
• Meeting play schedule of each medium
• If a SIU with a play instant time T,we must
  overcome Delay L then archive sync
• Define control time C,CgtL, RT - C

13
Buffer Configuration at the Client Side

• Dont consider clock drift problem first
• solve the problem of data overflow or underflow
  that caused by all random delays
• How to decide buffer size?
• Its simple case for intra-media sync
• Suppose we have a play schedule for a certain
  medium as shown in this graphic

14
Buffer Configuration at the Client Side

• Each SIU has the same duration P
• Two kind of delay
• Retrieval delay d1(fetch, divide media to SIU)
• Transmission delay d2(server to client)
• The time difference between data present time and
  arrival time is C-(d1d2)
• Its also buffering time
• The max buffer time is TbC-(d1mind2min)
• We can know d1mind2minTblt d1mind2minPN
• gt NgtTb/P ,N is also equal to frame number

15
Buffer Configuration at the Client Side

• If the max and min delay can be know in
  advance,the perfect buffer size is
  BDmax-Dmin/P
• Inter-media sync
• Decide multiple Inter-media buffer size is more
  complex (different delays,data size,server
  location)
• suppose there two media,audio and video
• Playback period is 2,Dminv1,Dmaxv5,Dmina1,Dmaxa
  7
• By above concept,Video perfect buffer size is 2,
• Audio perfect buffer size buffer is 3

16
Buffer Configuration at the Client Side
Based on Dmaxa7
Based on Dmaxv5
17
Buffer Configuration at the Client Side

• When two related media have different max
  delay,but are retrieve at the same time,then
  there is a constraint(over and underflow)
  according to each others max delay
• The medium retrieval schedule is a critical issue
  here
• Find a reasonable control time for independent
  media
• Ci(control time) is the skew between data
  retrieval and inter-media presentation gt
  CiTi-Ri

18
Determination of Control Time in Stored Data
Retrieval

• Ci dependent on retrieval delay(d1) and
  transmission delay(d2) of SIU,that we can say
  Cd1d2
• Decompose d1d1mind1var
• Decompose d2 (packet) Dp DtDv
• Dp(propagation delay), Dt(transmission delay),
  Dv(vary by network traffic)
• Rx/p,packet number of a media object
• d2 (object) Dp r Dt ?Dvr
• Cid1id2i is more accurate to estimate for
  schedule media retrieval time

19
Buffer Control System

• Problem still exist
• Dynamic frame size with depend heavily on
  compression method
• unexpected packet loss
• Clock drift
• Cause the continuous media gap

20
Buffer Control System

• Fundamental concept(1)-time tolerance
• The synchronization errors that can be tolerated
  by human perception vary in different application
  scenarios
• Use in inter-media sync to control parameter

21
Buffer Control System

• Fundamental concept(2)-Threshold
• The synchronization errors that can be tolerated
  by human perception vary in different application
  scenarios
• Define three buffer status
• Low threshold, Nominal threshold, high threshold

22
Buffer Control System

• Fundamental concept(2)-Threshold
• Control action
• LHduplicating current frame with skew tolerance
• NHsmoothly play
• HH discard frame with skew tolerance
• must be some information loss but it can be
  tolerance
• LH and HHs setting influence performance
• LH and HH are too closed ends
• Data overflow and underflow reduce less
• LH and HH are so far away from ends
• Too frequent control action to cause information
  loss

23
Buffer Control System

• Fundamental concept(3)-Nominal value
• The same concept but new definition in Nominal
  value
• Nm is current frame count of one medium in its
  corresponding buffer
• HTm and LTm are the thresholds in two ends of the
  buffer for this medium

24
Buffer Control System

• Fundamental concept(3)-Nominal value
• To solve the problem
• when each medium has its buffer,buffer over-
  and underflow, frame discard or duplication
  independently,but result in frame index
  inconsistency beyond skew tolerance
• Master and slave medium method
• Choose important one is master medium and another
  is slave medium
• adding this difference value to the current slave
  nominal value until both medium frame indexes is
  the same

25
Buffer Control System

• Buffer control scheme
• Comparator
• Regulator
• Corrector
• Buffer auto control model
• Switch mode
• First-order integrator
• Second-order integrator

26
Simulation

• The network and operation system delays is 10-100
  ms by reference 3
• Test with control time 300ms (gtD)
• Play period is 33 ms
• Buffer size is 10 according to C/P
• LH and HH are setting to be 2 and 8
• Delay distribution is Gaussian or exponential
  from 50ms to 500ms
• A total of 100,000 framed are tested

27
Output(1)
28
Output(2)
29
Output(3)
30
Output(4)
31
Output(5)
32
Output(6)
33
Result of Observation

• Dynamic frame size does not have much effect on
  the buffer control scheme
• Dynamic frame sizes under- and overflow counts
  more than constant frame on the no control
• Buffer overflow and underflow caused by clock
  drift is less frequent compared to delay
• when buffer overflow and underflow are dominated
  by delay jitter, the clock drift is not a problem
  in inter-media sync
• With buffer control scheme, the problem caused by
  clock drift also be eliminated

34
Result of Observation

• Output6(in Table15,16)show that disruption
• has been effected by buffer frame size
• The number of buffer overflows and underflows are
  greatly decreased due to this relatively perfect
  buffer size configuration even without any
  control method
• The simulation results show that with the perfect
• buffer size configuration and a suitable buffer
  control method applied to the receivers buffer,
  media synchronization can be achieved.

35
Conclusion

• A suitable buffer size is a critical issue to
  reach sync(dependent on delay)
• Buffer control scheme is need when estimated
  delays do match buffer size
• How to better predict buffer under- and overflow
• A improved buffer control scheme can be more
  efficient
• a model to describe the real world delay jitter
  more exactly