Spatio-Temporal Frequency Analysis for Removing Rain and Snow from Videos

1
Spatio-Temporal Frequency Analysis for Removing
Rain and Snow from Videos
Peter Barnum Takeo Kanade Srinivasa Narasimhan
Carnegie Mellon University
June 16, 2007
2
Bad weather in outdoor videos
Rain
Snow
3
Previous Work
Image-based blurring
Camera-based blurring
Spikes due to rain
Pixel Intensity
Garg and Nayar ICCV 05
Streak detection
Time
Hase et al. ICIP 98 Starik and Werman IWTAS
03 Zhang et al. ICME 06
Garg and Nayar CVPR 04
4
Groups of streaks
5
Imaging a falling particle
Gaussians
Snowflakes
Raindrops
Breadth
6
Gaussian model of streak appearance
Camera parameters are constant
A Gaussian
A blurred Gaussian streak
Including streak orientation (just a coordinate
space rotation)
7
Where are the streaks?
8
Fourier transform of the streaks
9
Building a complete model
For a given precipitation intensity
For all common drop sizes
Blurred Gaussian
For all depths that are in-focus
10
Model accuracy
Original image
2D Fourier Transform
Model
Model with 50 randomly set to zero
11
Finding the precipitation rate
Rain and snow have two useful properties
Mailbox
Building
100
0
Snow
12
Frame-to-frame differences
t1
t3
t2
w-1
w1
w0
13
Frame-to-frame differences
t1
t3
t2
w-1
w1
w0
14
Finding the precipitation rate
For most objects
But for rain and snow
Because of these properties
15
Estimating the model parameters
Precipitation rate
Streak orientation
The precipitation rate is approximately
Estimating the streak orientation requires a
spatial consistent estimate
The orientation is found by
16
Frequency space examples
Original image
2D FT
Model
17
Computing per-frequency estimates
At a given frequency
18
Computing per-pixel estimates
19
Refining the single frame estimate
Exactly the same model, constant in w
20
Computing per-pixel estimates
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Conclusions and future work
A global frequency method for rain and snow
removal
Refining global estimates with local features
Into each life some rain must fall. -Henry
Wadsworth Longfellow
25
Extra slides
26
Imaging a falling particle