Image inpainting with sparse prior

1
Image inpainting with sparse prior

• Spencer Brady
• ECE 544 Project
• 11/30/2007

2
Outline

• Introduction
• Definition of Inpainting
• Problem Statement
• Simplifying Assumptions
• Results of Different Approaches
• DCT
• Separable 9-7 Wavelets
• Contourlets
• Morphological Component Analysis
• PSNR Comparisons

3
Introduction

• Inpainting estimates the missing pixels based on
  some prior image model

Inpainting
Original Image
Reconstructed Image
4
General Problem Statement
We model the true image (with no missing pixels)
as having a sparse representation in T where
5
General Problem Statement
We wish to solve the following optimization
problem
From which we recover the inpainted image
6
Simplifying Assumptions

• Replace l0 norm with l1 norm
• Replace coefficients with inverse transform
  ,
• then do the minimization over instead of

7
Simplified Problem Statement
Now we minimize over the set of images instead
of the set of coefficients
The inpainted image is nothing but
8
How do we solve this?

• Recall
• Has a decoupled solution given by soft
  thresholding

ai
bi
d
-d
9
General Algorithm

• Iterative Soft Thresholding
• Initialize Parameters (threshold d, iterations N,
  xhat)
• Loop N times
• (a) Decompose/Threshold/Reconstruct
• - Compute transform of M(x)(1-M)xhat
• - Soft threshold the coefficients with d
• - Reconstruct xhat from thresholded coefficients
• 3) Set output to of M(x)(1-M)xhat

10
DCT Approach

• Choose
• T the 2-D Discrete Cosine Transform

11
DCT Results
N40 d6
PSNR 40.64
12
DCT Results
N40 d10
PSNR 31.11
13
DCT Results
N40 d35
PSNR 34.04
14
DCT Results
N50 d60
PSNR 30.00
15
DCT Results
N80 d250
PSNR 22.36
16
DCT Results
N60 d28
PSNR 32.98
17
DCT Results
N60 d25
PSNR 35.21
18
DCT Results
N40 d15
PSNR 42.18
19
Separable Wavelet Approach

• Choose
• T separable 2-D Wavelet Transform
• Cohen-Daubechies-Feauveau biorthogonal 9-7
  wavelet with 7 levels of decomposition

20
Separable Wavelet Results
N60 d15
PSNR 36.61
21
Separable Wavelet Results
N80 d15
PSNR 27.76
22
Separable Wavelet Results
N70 d15
PSNR 34.12
23
Separable Wavelet Results
N70 d20
PSNR 29.93
24
Separable Wavelet Results
N150 d100
PSNR 21.69
25
Separable Wavelet Results
N80 d30
PSNR 31.33
26
Separable Wavelet Results
N90 d15
PSNR 33.81
27
Separable Wavelet Results
N90 d10
PSNR 41.91
28
Contourlet Approach

• Choose
• T the Contourlet Transform
• 5 4 4 3 3 levels/directions
• pkva filters

29
Contourlets Results
N40 d5
PSNR 39.71
30
Contourlets Results
N50 d10
PSNR 27.62
31
Contourlets Results
N90 d10
PSNR 36.13
32
Contourlets Results
N90 d10
PSNR 32.38
33
Contourlets Results
N70 d40
PSNR 15.79
34
Contourlets Results
N70 d20
PSNR 33.13
35
Contourlets Results
N70 d8
PSNR 31.22
36
Contourlets Results
N40 d10
PSNR 42.10
37
Morphological Component Analysis Approach
Cartoon component
Texture component
38
Sparse Image Model


Cartoon component
Texture component
Images obtained from 1
39
Problem Statement
We wish to solve the following optimization
problem
From which we obtain the inpainted image
40
Simplified Problem Statement
Now we minimize over a set of images rather than
a set of coefficients (reducing the
dimensionality of the problem)
From which the inpainted image simply becomes
41
How do we solve this?

• Recall
• Has the decoupled solution of soft thresholding

ai
bi
d
-d
42
Proposed Algorithm

• Iterative Soft Thresholding
• Initialize Parameters (threshold d, iterations N)
• Loop N times
• (a) Update xn
• - Compute contourlet transform of M(x-xt)(1-M)
  xn
• - Soft threshold the coefficients with d
• - Reconstruct xn from thresholded coefficients
• (b) Update xt
• - Compute local DCT transform of M(x-xn)(1-M)
  xt
• - Soft threshold the coefficients with d
• - Reconstruct xt from thresholded coefficients
• 3) Set output to xt xn

43
MCA Results
N40 d5
PSNR 41.24
44
MCA Results
N40 d5
PSNR 30.44
45
MCA Results
N60 d30
PSNR 32.52
46
MCA Results
N60 d60
PSNR 23.07
47
MCA Results
N40 d5
PSNR 14.14
48
MCA Results
N40 d5
PSNR 26.12
49
MCA Results
N40 d5
PSNR 29.12
50
MCA Results
N40 d5
PSNR 45.93
51
PSNR Comparisons
52
Comment
53
References

• 1 Elad, Starck, Querre, Donoho, Simultaneous
  cartoon and texture image inpainting using
  morphological component analysis, Applied and
  Computational Harmonic Analysis, 19 (2005)
  340-358
• 2 Starck, Elad, Donoho, Image decomposition
  via the combination of sparse representations and
  a variational approach, IEEE Transactions on
  Image Processing, Oct 2005
• 3 Rudin, Osher, Fatemi, Nonlinear total
  variation based noise removal algorithms,
  Physica D 60 (1992) 259-268
• 4 Sardy, Bruce, Tseng, Block coordinate
  relaxation methods for nonparametric signal
  denoising with wavelet dictionaries, Journal of
  Computational and Graphical Statistics (2000)
  361-379
• 5 Donoho, Ideal spatial adaptation by wavelet
  shrinkage, Biometrica (1994) 81, 3, pp.425-55
• 6 Do, Vetterli, The contourlet transform an
  efficient directional multiresolution image
  representation, IEEE Transactions on Image
  Processing, vol. 14, no. 12, pp. 2091-2106, Dec.
  2005

54
Questions