RealTime Oblivious Erasure Correcting

1
Real-Time Oblivious Erasure Correcting

• Amos Beimel
• Shlomi Dolev
• Noam Singer

2
Motivation

• Servers with large files
• Many downloading client
• Non reliable communication

3
Motivation (cont)

• Problem
• Channels with high loss rate.
• Expensive feed-back channels
• Current solutions
• ARQ Requires large feed-back
• FEC High decoding complexity
• Our innovation
• Meeting them half way

4
Background

• FEC
• Previous works
• rate-less codes
• Real-time codes
• Our contribution

5
FEC Forward Error Correction
6
Previous works

• I.S.Reed, G.Solomon - 1960
• Optimal efficiency
• Quadratic complexity
• Tornado Codes / M.G.Luby, M.Mitzenmacher,
  M.A.Shokrollahi, D.Spielman - 2001
• Efficiency (1e)k
• Linear complexity
• Graph construction

7
Previous works (cont)

• LT Codes / M.G.Luby - 2002
• Efficiency
• Complexity O(n log n)
• Online Codes / Peter Maymounkov - 2002
• Efficiency (1e)k
• Linear complexity

8
Rate-less Codes

• No predetermined channel rate Rk/n
• Can basically send 8 encodings
• Can decode the message from any large enough
  subset of symbols.
• Rate-less RS, LT Codes, Online Codes
• Non rate-less Tornado Codes

9
Real-Time Codes

• Complexity
• Encoding a symbol
• Reception of a symbol
• Decoding the entire message
• Decoding rate
• How often do we decode a symbol
• Decoded prefix
• Stalling at the end of the message

10
Our contribution

• Combine ARQ with Erasure correcting
• Rate-Less
• Real-Time
• Low decoder memory overhead
• Reduced feed-back
• Efficiency O(k)
• Complexity O(k log k)

11
Our protocol

• Protocol description
• Protocol properties
• Decoding probabilities
• Handling special cases

12
Oblivious decoding

• Each encoding is XORed from some input symbols.
• Decoding is made if exactly one symbol is
  missing.
• Otherwise, encoding is dumped.

13
Decoding
Encodings
Data Symbols
14
Protocol description

• Calculate degree m
• Pick m symbols
• XOR them
• Transmit encoding

• Check if exactly 1 symbol missing
• If not, dump
• Otherwise decode
• Transmit decoded count

Encodings
m3
Feed-back
d3
15
Protocol properties

• Low memory overhead
• Missing bitmap of size k
• Trivial encoder/decoder
• Rate-Less
• Real-Time

16
Decoding probability P(m,d,k)

• Conventions
• m Encoding degree
• d Decoded symbols count
• Decoding only if exactly one symbol is missing

17
P(m,d,k) samples

• P(m1,d0,k) 1
• P(mk,dk-1,k) 1
• P(m1,d,k) (k-d)/k
• P(m1,dk/2,k) P(m2,dk/2,k) 0.5
• P(mgtd1,d,k) 0

18
Decoding probability P(m,d,k)

• There are m possible missing symbols
• Each can be from the (k-d) missing symbols
• The rest of the (m-1) are from the remainingd
  out of (k-1), (d-1) out of (k-2),

19
P(m,d,k) for m1
20
P(m,d,k) for m2
21
P(m,d,k) for each m
22
P(m,d,k) maximization on m
23
Maximization using m

• For each d the optimal degree m, to which
  P(m,d,k) is maximized.
• P(m-1,d,k) P(m,d,k) P(m-1,d,k)
• Result

24
Optimal degree m(d,k)
25
Degree changes

• For ?
• There are different values for m
  until
• Remaining values of d possible
  different values for m
• May achieve feed-back messages

26
Optimal probability

• Decoding probability for using optimal degree
• Lemma

27
Optimal probability gt1/e
28
Decoding rate

• After expected a decoding occur
• Expected less than e symbols
• Total expected required encodings

29
Required encodings (graph)
30
Decoding rate (graph)
encodings
5 experiments
31
Decoding Rate (LT Codes)
encodings
5 experiments
32
Expected required encodings

• exp Expected total symbols required
• exp e?k
• We will later show exp 2?k
• Experiments show expk?1.856

33
Proof (explt2k)

• Assume probability lt P(d,k)

34
The last symbols

• The degree m changes on each decoding.
• Feed-back delays reduces decoding efficiency
• Experiments show
• We may use the same degree of
• Expected required encodings

35
Conclusion - protocol properties

• Low memory overhead
• Missing bitmap of size k
• Trivial encoder/decoder
• Rate-Less
• Real-Time
• Low feed-back messages
• Robustness with feed-back losses
• Decoding rate gt 1/e
• Efficiency k?1.856
• Complexity O(k log k)

36
Heuristic enhancements

• First k encodings
• Estimating d

37
First k encodings

• Sender sends first symbols as is
• Another option
• Send a permutation
• May resolve loss bursts
• Efficient for higher channel rate R
• exp Rk(1-R)ke k(e-R(e-1))
• Non rate-less

38
Estimating d

• Feed-back messages are delayed or lost
• Inaccurate knowledge of d, reduces process
  efficiency.
• Server may estimate d using
• Assuming constant channel rate R
• Estimate d from the global decoding rate.
• Estimate d using a local adaptive estimation.

39
Application

• Smooth non-sequential transfer
• ARQ enhancement
• Broadcast over a tree structure
• Multi source download / immigration

40
Smooth non-sequential transfer

• Ordered transfer may not be required
• Data base items
• Pictures (non-compressed)
• MPEG
• Partial message may be enough
• Commercials
• News updates
• Ordering can be postponed
• Decoding rate is at least 1/e

41
ARQ enhancement

• Used on retransmitting of a window
• Sending encodings instead of the entire window
• Feed-back messages
• Continuous decoded prefix
• Decoded symbols count on window
• May reduce feed-back messages
• Robustness with feed-back losses
• Useful with low channel rates

42
Broadcast over a tree structure

• Encodings sent from ancestors to children
• Feed-back (d) replied to ancestors
• Children cant know more than ancestors
• Each middle host forwards encodings or encodes
  from its decoded set

43
Multi source download / immigration

• Client may communicate with several servers
• To each it send its d
• Each server generates independent encodings
• Servers crash/loss would not interrupt transfer

44
The End

• Thank you