Learning to Perceive Transparency from the Statistics of Natural Scenes

1
Learning to Perceive Transparency from the
Statistics of Natural Scenes

• Anat Levin
• School of Computer Science and Engineering
• The Hebrew University of Jerusalem

Joint work with Assaf Zomet and Yair Weiss
2
Transparency
3
How does our visual system choose the right
decomposition?

• Why not simpler one layer solution?
• Which two layers out of infinite possibilities?

4
Talk Outlines

• Motivation and previous work
• Our approach
• Results and future work

5
Transparency in the real world
Fashion Planet's photographers have spent the
last five years working to bring you clean
photographs of the windows on New York especially
without the reflections that usually occur in
such photography
http//www.fashionplanet.com/sept98/features/refle
ctions/home
6
Transparency and shading
7
Transparency in human vision
Two layers
One layer

• Metelli's conditions (Metelli 74)
• T-junctions, X-junctions, doubly reversing
  junctions (Adelson and Anandan 90, Anderson 99)

Not obvious how to apply junction catalogs to
real images.
8
Transparency from multiple frames

• Two frames with polarizer using ICA (Farid and
  Adelson 99, Zibulevsky 02)
• Multiple frames with specific motions (Irani et
  al. 94, Szeliski et al. 00, Weiss 01)

9
Shading from a single frame

• Retinex (Land and McCann 71).
• Color (Drew, Finlayson Hordley 02)
• Learning approach (Tappen, Freeman Adelson 02)

10
Talk Outlines

• Motivation and previous work
• Our approach
• Results and future work

11
Our Approach
Ill-posed problem. Assume probability
distribution Pr(I1), Pr(I2) and search for most
probable solution.
(ICA with a single microphone)
12
Statistics of natural scenes
Input image
dx histogram
dx Log histogram
13
Statistics of derivative filters
0
Gaussian x2
Log Probability
Laplacian x
x 1/2
-1
Log histogram
Generalized Gaussian distribution (Mallat 89,
Simoncelli 95)
14
Is sparsity enough?
Or
15
Is sparsity enough?


Or


Exactly the same derivatives exist in the single
layer solution as in the two layers solution.
16
Beyond sparseness

• Higher order statistics of filter outputs (e.g.
  Portilla and Simoncelli 2000).
• Marginals of more complicated feature detectors
  (e.g. Zhu and Mumford 97, Della Pietra Della
  Pietra and Lafferty 96).

17
Corners and transparency

• In typical images, edges are sparse.
• Adding typical images is expected to increase the
  number of corners.
• Not true for white noise

18
Harris-like operator
19
Corner histograms
Derivative Filter
Corner Operator
20
Fitting
Derivative Filter
Corner Operator
Typical exponents for natural images
21
Simple prior for transparency prediction
The probability of a decomposition
22
Does this predict transparency?
23
How important are the statistics?
Is it important that the statistics are non
Gaussian? Would any cost that penalized high
gradients and corners work?
24
The importance of being non Gaussian
25
The scalar transparency problem
Consider a prior over positive scalars For
which priors is the MAP solution sparse?
26
The scalar transparency problem
Observation The MAP solution is obtained with
a0, b1 or a1, b0 if and only if f(x)log
P(x) is convex.
0
1
0
1
0.5
0.5
MAP solution a0, b1
MAP solution a0.5, b0.5
27
The importance of being non Gaussian
28
Can we perform a global optimization?
29
Conversion to discrete MRF
For the decomposition
gradient at location i
Local Potential- derivative filters
Pairwise Potential- pairwise approximation to the
corner operator
-Enforcing integrability
30
Conversion to discrete MRF
For the decomposition
Local Potential- derivative filters
Pairwise Potential- pairwise approximation to the
corner operator
-Integrability enforcing
31
Optimizing discrete MRF
possible assignments. Solution use
max-product belief propagation. The MRF has many
cycles but BP works in similar problems (Freeman
and Pasztor 99, Frey et al 2001. Sun et al 2002).
Converges to strong local minimum (Weiss and
Freeman 2001)
32
Drawbacks of BP for this problem

• Large memory and time complexity.
• Convergence depends on update order.
• Discretization artifacts

33
Talk Outlines

• Motivation and previous work
• Our approach
• Results and future work

34
Results
input
Output layer 1
Output layer 2
35
Results
input
Output layer 1
Output layer 2
36
Future Work
37
Future Work

• Dealing with a more complex texture

• Applying other optimization methods.

38

• Conclusions
• Natural scene statistics predict perception of
  transparency.
• First algorithm that can decompose a single image
  into the sum of two images.