Improving analysis and performance of modern error-correction schemes

1
Improving analysis and performance of modern
error-correction schemes a physics approach
UoC, 04/10/06
Misha Chertkov (Theory Division, LANL) Vladimir
Chernyak (Department of Chemistry, Wayne
State) Misha Stepanov (Theory Division,
LANL) Bane Vasic (Department of ECE, University
of Arizona)
Analyzing error-floor for LDPC codes
Understanding Belief-Propagation Loop Calculus
Phys.Rev.Lett. 93, 198702 (2004) IT workshop, San
Antonio 10/2004 CNLS workshop, Santa Fe
01/2005 Phys.Rev.Lett. 95, 228701
(2005) arxiv.org/abs/cond-mat/0506037 arxiv.org/ab
s/cs.IT/0507031 IT workshop, Allerton
09/2005 arxiv.org/abs/cs.IT/0601070 arxiv.org/abs/
cs.IT/0601113
MC,VC
arxiv.org/abs/cond-mat/0601487 arxiv.org/abs/cond-
mat/0603189
towards improving Belief Propagation
2

• Analogous vs Digital
• Analogous Error-Correction
• Digital Error-Correction
• LDPC, Tanner graph, Parity Check
• Inference, Maximum-Likelihood, MAP
• MAP vs Belief Propagation (sum-product)
• BP is exact on the tree
• Error-correction Optimization
• Shannon-Transition
• Error-floor

Menu (first part)
Introduction

• Instanton method the idea
• Instanton-amoeba (efficient numerical method)
• Test code (155,64,20) LDPC
• Instantons for the Gaussian channel (Results)
• BER Monte-Carlo vs Instanton

Instanton proof of principles test

• Conclusions
• Path Forward

3
Analogous vs digital
Analogous
Digital
discrete easy to copy
continuous hard to copy
0111100101
camera picture music on tape
typed text computer file
real number better/worse
integer number yes/no
menu
4
Error-correction for analogous
One iteration
menu
5
menu
6
menu
7
Digital Error-Correction
N gt L RL/N - code rate
Coding
Decoding
noise
white
channel
example
Gaussian symmetric
menu
8
(linear coding)
Low Density Parity Check Codes
N10 variable nodes
Parity check matrix
Tanner graph
MN-L5 checking nodes
spin variables -
- set of constraints
menu
9
Parity check matrix (155,64,20) code
Tanner graph (155,64,20) code
menu
10
Inference
Maximum-Likelihood (ML) Decoding
menu
11
Decoding (optimal)
magnetic field log-likelihood
magnetizationa-posteriori log-likelihood
menu
12
Sub-optimal but efficient decoding
Belief Propagation (BPsum-product)
Gallager63Pearl 88MacKay 99
solving Eqs. on the graph
Iterative solution of BP Message Passing (MP)
QmN steps instead of Q - number
of MP iterations m - number of checking nodes
contributing a variable node
What about efficiency? Why BP is a good
replacement for MAP?
menu
(no loops!)
13
Tree -- no loops -- approximation
Analogy Bethe lattice (1937)
MAP
BP
Belief Propagation is optimal (i.e. equivalent to
Maximum-A-Posteriori decoding) on a tree (no
loops)
Gallager 63 Pearl 88 MacKay 99 Yedidia,
Freeman, Weiss 01
menu
14
Bit Error Rate (BER)
Probability of making an error in the bit i
probability density for given magnetic
field/noise realization (channel)
measure of unsuccessful decoding
Digital error-correction scheme/optimization

• describe the channel/noise --- External
• suggest coding scheme
• suggest decoding scheme
• measure BER/FER
• If BER/FER is not satisfactory (small enough)
  goto 2

menu
15
Shannon transition/limit
BER, B
SNR, s
menu
From R. Urbanke, Iterative coding systems
16
Error floor (finite size BP-approximate)
Error floor prediction for some regular (3,6)
LDPC Codes using a 5-bit decoder. From T.
Richardson Error floor for LDPC codes, 2003
Allerton conference Proccedings.
No-go zone for brute-force Monte-Carlo
numerics. Estimating very low BER is the major
bottleneck in coding theory/practice
menu
17
Our (current) objective For given (a) channel
(b) coder (c) decoder to estimate BER by means
of analytical and/or semi-analytical methods.
Hint BER is small and it is mainly formed at
some very special bad configurations of the
noise/magnetic field Instanton approach is the
right way to identify the bad configurations
and thus to estimate BER!
menu
18
Instanton Method
Laplace method Saddle-point method Steepest
descent
menu
19
Parity check matrix (155,64,20) code
Tanner graph (155,64,20) code
menu
20
Found with numerical instanton-amoeba scheme
menu
instanton-amoeba
21
Instantons for (155,64,20) code Gaussian channel
Phys. Rev. Lett -- Nov 25, 2005
menu
22
menu
23
Conclusions (for the first part error floor
analysis)
We suggested amoeba-instanton method for
efficient numerical evaluation of BER in the
regime of high SNR (error floor). The main idea
error-floor is controlled by only a few most
damaging configurations of the noise
(instantons).
Results of the amoeba-instanton are successfully
validated against brut-force Monte-Carlo (in the
regime of moderate SNR)
menu
24
Path Forward
Inter-symbol interference noise (2d and 3d
error-correction) Distributed coding, Network
coding Combinatorial optimization
menu
25
Understanding Belief Propagation

• Questions
• Why it works so well even when it should not?
  -- BP is gauge fixing condition
• Can one constructs a full solution (MAP) from
  BP? -- yes one can!/loop series

Answers arxiv.org/abs/cond-mat/0601487 arxiv.org/
abs/cond-mat/0603189

• Making use of the loop calculus/series
• Improving BP approximate algorithms

• LDPC decoding
• SATisfiability resolution
• Data reconstruction
• Clustering
• etc

first slide
26
Ising variables on edges
Vertex Model
Partition function
Probability
Reduction to bipartite graph (error-correction)
improving BP
27
Bethe Free Energy --- Variational Approach
self-energy
entropy
entropy correction
Constraints (introduce in minimization through
Lagrange multipliers)
Belief Propagation (Bethe-Peierls) equations
Generalization of Yedidia, Freeman,Weiss 01
improving BP
28
Loop series
C
--- beliefs (prob.) calculated within BP !

• BP is special, not only without loops!
• Gauge invariant representation!

Three alternative derivations

• integral representation
• algebraic representation
• gauge representation

improving BP
29
Loop series (derivation 1)
? vertex
propagator ?
improving BP
30
Loop series (derivation 2)
Expand the vertex (edge) term

• Each node enters the product only once
• Node is colored if it contains at least
• one colored edge

Calculate resulting terms one-by-one


Gauge fixing condition
To forbid
loose end contribution for any node !!
improving BP
31
Loop series (derivation 3)
fixing the gauge!! to kill loops
equations
Belief Propagation !!
Loop series has just been derived!!
improving BP
32

• Conclusions ( for the second part
  Understanding/Improving BP)
• Loopy BP works well because
• BP is nothing
  but GAUGE FIXING condition
• Simple finite series --- LOOP SERIES ---
• for MAP is
  constructed in terms of BP solution

Future work

• Approximate algorithms
• --- leading loop,
  next after leading,..
• --- apply to LDPC
  decoding
• --- different
  graphs, lattices
• Generalization
• --- Ising ? Potts
  (longer alphabets)
• --- continuous
  alphabets
• 
  (XY,Heisenberg,Quantum models)

first slide
improving BP
33
(No Transcript)
34
Instantons on the tree (semi-analytical)
m2, l3, n3
m3, l5, n2
ITW 2004, San Antonio
PRL 93, 198702 (2004)
menu
35
Instanton-amoeba (efficient-numerical scheme)
unite vector in the noise space
error-surface
To minimize BER with respect to the unit vector !!
Minimization method of our choice is
simplex-minimization (amoeba)
menu
instanton-amoeba for Tanner code
36
Different noise models for different channels
Linear
simplifications
White
Symmetric
Gaussian
Laplacian
menu
37
Phys.Rev.Lett. 95, 228701 (2005)
Rational structure of instanton (computational
tree analysis/explanation)
min-sum
4 iterations
Minimize effective action keeping the condition
menu
based on Wiberg 96
38
Bit-Error-Rate Gaussian channel
menu
39
Instantons for (155,64,20) code Laplacian channel
menu
IT workshop, Allerton 09/2005
40
Instantons as medians of pseudo-codewords
PRL -- Nov 25, 2005
menu
41
Bit-Error-Rate Laplacian channel
menu