Sparse Recovery Using Sparse Random Matrices

1
Sparse Recovery Using Sparse (Random) Matrices

• Piotr Indyk
• MIT

Joint work with Radu Berinde, Anna Gilbert,
Howard Karloff, Martin Strauss and Milan Ruzic
2
Goal of this talk

• Stay awake until the end

3
Linear Compression(learning Fourier coeffs,
linear sketching, finite rate of innovation,
compressed sensing...)

• Setup
• Data/signal in n-dimensional space x
• E.g., x is an 1000x1000 image ?
  n1000,000
• Goal compress x into a sketch Ax ,
• where A is a m x n sketch matrix, m ltlt n
• Requirements
• Plan A want to recover x from Ax
• Impossible undetermined system of equations
• Plan B want to recover an approximation x of
  x
• Sparsity parameter k
• Want x such that x-xp? C(k) x-xq
  ( lp/lq guarantee )
• over all x that are k-sparse (at most k
  non-zero entries)
• The best x contains k coordinates of x with the
  largest abs value
• Want
• Good compression (small m)
• Efficient algorithms for encoding and recovery
• Why linear compression ?

4
Linear compression applications

• Data stream algorithms
• (e.g. for network monitoring)
• Efficient increments
• A(x?) Ax A?
• Single pixel camera
• Wakin, Laska, Duarte, Baron, Sarvotham,
  Takhar, Kelly, Baraniuk06
• Pooling, Microarray Experiments Kainkaryam,
  Woolf, Hassibi et al, Dai-Sheikh, Milenkovic,
  Baraniuk

5
Constructing matrix A

• Choose encoding matrix A at random
• (the algorithms for recovering x are more
  complex)
• Sparse matrices
• Data stream algorithms
• Coding theory (LDPCs)
• Dense matrices
• Compressed sensing
• Complexity/learning theory
• (Fourier matrices)
• Traditional tradeoffs
• Sparse computationally more efficient, explicit
• Dense shorter sketches
• Goal the best of both worlds

6
Prior and New Results
7
Prior and New Results
Excellent
Scale
Very Good
Good
Fair
8
Recovery in principle (when is a matrix good)
9
dense vs. sparse

• Restricted Isometry Property (RIP) - sufficient
  property of a dense matrix A
• ? is k-sparse ? ?2? A?2 ? C ?2
• Holds w.h.p. for
• Random Gaussian/Bernoulli mO(k log (n/k))
• Random Fourier mO(k logO(1) n)
• Consider random m x n 0-1 matrices with d ones
  per column
• Do they satisfy RIP ?
• No, unless m?(k2) Chandar07
• However, they can satisfy the following RIP-1
  property Berinde-Gilbert-Indyk-Karloff-Strauss08
  
• ? is k-sparse ? d (1-?) ?1? A?1 ?
  d?1
• Sufficient (and necessary) condition the
  underlying graph is a
• ( k, d(1-?/2) )-expander

Other (weaker) conditions exist, e.g., kernel
of A is a near-Euclidean subspace of l1. More
later.
10
Expanders

• A bipartite graph is a (k,d(1-?))-expander if for
  any left set S, Sk, we have N(S)(1-?)d S
• Constructions
• Randomized mO(k log (n/k))
• Explicit mk quasipolylog n
• Plenty of applications in computer science,
  coding theory etc.
• In particular, LDPC-like techniques yield good
  algorithms for exactly k-sparse vectors x
• Xu-Hassibi07, Indyk08, Jafarpour-Xu-Hassibi-Ca
  lderbank08

N(S)
d
S
m
n
11
Proof d(1-?/2)-expansion ? RIP-1

• Want to show that for any k-sparse ? we have
• d (1-?) ?1? A ?1 ? d?1
• RHS inequality holds for any ?
• LHS inequality
• W.l.o.g. assume
• ?1 ?k ?k1 ?n0
• Consider the edges e(i,j) in a lexicographic
  order
• For each edge e(i,j) define r(e) s.t.
• r(e)-1 if there exists an edge (i,j)lt(i,j)
• r(e)1 if there is no such edge
• Claim 1 A?1 ?e(i,j) ?ire
• Claim 2 ?e(i,j) ?ire (1-?) d?1

d
m
n
12
Recovery algorithms
13
Matching Pursuit(s)
A
x-x
Ax-Ax


i

i

• Iterative algorithm given current approximation
  x
• Find (possibly several) i s. t. Ai correlates
  with Ax-Ax . This yields i and z s. t.
• xzei-xp ltlt x - xp
• Update x
• Sparsify x (keep only k largest entries)
• Repeat
• Norms
• p2 CoSaMP, SP, IHT etc (dense matrices)
• p1 SMP, this paper (sparse matrices)
• p0 LDPC bit flipping (sparse matrices)

14
Sequential Sparse Matching Pursuit

• Algorithm
• x0
• Repeat T times
• Repeat SO(k) times
• Find i and z that minimize A(xzei)-b1
• x xzei
• Sparsify x
• (set all but k largest entries of x to 0)
• Similar to SMP, but updates done sequentially

A
i
N(i)
Ax-Ax
x-x
Set zmedian (Ax-Ax)N(i) . Instead, one
could first optimize (gradient) i and then z
Fuchs09
15
SSMP Running time

• Algorithm
• x0
• Repeat T times
• For each i1n compute zi that achieves
• Diminz A(xzei)-b1
• and store Di in a heap
• Repeat SO(k) times
• Pick i,z that yield the best gain
• Update x xzei
• Recompute and store Di for all i such that
• N(i) and N(i) intersect
• Sparsify x
• (set all but k largest entries of x to 0)
• Running time
• T n(dlog n) k nd/md (dlog n)
• T n(dlog n) nd (dlog n) T nd (dlog
  n)

A
i
Ax-Ax
x-x
16
SSMP Approximation guarantee

• Want to find k-sparse x that minimizes x-x1
• By RIP1, this is approximately the same as
  minimizing Ax-Ax1
• Need to show we can do it greedily

17
Experiments
256x256
SSMP is ran with S10000,T20. SMP is ran for 100
iterations. Matrix sparsity is d8.
18
Conclusions

• Even better algorithms for sparse approximation
  (using sparse matrices)
• State of the art can do 2 out of 3
• Near-linear encoding/decoding
• O(k log (n/k)) measurements
• Approximation guarantee with respect to L2/L1
  norm
• Questions
• 3 out of 3 ?
• Kernel of A is a near-Euclidean subspace of l1
  Kashin et al
• Constructions of sparse A exist
  Guruswami-Lee-Razborov, Guruswami-Lee-Wigderson
• Challenge match the optimal measurement bound
• Explicit constructions

Thanks!
This talk