Webcamsynopsis: Peeking Around the World

1
Webcam-synopsisPeeking Around the World

• Young Ki Baik (CV Lab.)
• 2008. 4. 4 (Fri)

2
Webcam-Synopsis Peeking Around the World

• References
• Webcam Synopsis Peeking Around the World
• Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel
  Peleg
• (ICCV 2007)
• Non-Chronological Video Synopsis and Indexing
• Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI
  2008)
• Background Cut
• J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV
  2006)

3
Webcam-Synopsis Peeking Around the World

• What is Video-Synopsis?

Video synopsis is compact video data reducing
spatio-temporal redundancy in input video.
4
Webcam-Synopsis Peeking Around the World

• Demo (Final result)

5
Webcam-Synopsis Peeking Around the World

• How can we make Video Synopsis?
• Basic concept
• Detect activity tubes
• in input video I.
• Put activity tubes
• to the S
• considering relation between
• - tubes and background.
• - tube and other tubes.

6
Webcam-Synopsis Peeking Around the World

• Synopsis
• Properties of video synopsis S
• The video synopsis S should be substantially
  shorter than the original video I.
• Maximum activity from the original video should
  appear in the synopsis video.
• The dynamics of the objects should be preserved
  in the synopsis video. (for example, regular
  fast-forward may fail to preserve the dynamics of
  fast objects.)
• Visible seams and fragmented objects should be
  avoided.

7
Webcam-Synopsis Peeking Around the World

• OBJECT-BASED SYNOPSIS

8
Webcam-Synopsis Peeking Around the World

• Object-based synopsis construction
• Online Phase
• This phase is done in real time during video
  capture.
• Preparing step for response phase.
• Response Phase
• started with user query.
• This phase may take a few minutes, depending on
  the activity in the time period of interest.

9
Webcam-Synopsis Peeking Around the World

• Overall flow

10
Webcam-Synopsis Peeking Around the World

• ONLINE PHASE

11
Webcam-Synopsis Peeking Around the World

• Video Acquisition
• Video Volume Data

t
y
I(x, y, t) 3D Space Time Volume
(x, y) Spatial coordinates of
the image t Frame index
x
12
Webcam-Synopsis Peeking Around the World

• Background Video Construction
• Temporal Median

t
Temporal Window
y
Sorting by intensity
Selecting median value
x
13
Webcam-Synopsis Peeking Around the World

• Background Video Construction

t
t
y
y
x
x
I
B
14
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Using Background Cut (ECCV 2006)
• Using min-cut to get a smooth segmentation of
  foreground objects.
• Only considering 2D image (not video).

15
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Extracting moving objects
• We have information for

I(x, y, t) 3D Space Time Volume
B(x, y, t) 3D Space Time Volume for background
Current
Current image I (x, y,
current time)
Current background image
B (x, y, current time)
16
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Simplify Object detection problem to
• Binary labeling f Foreground 1, Background
  0
• Obtaining the desirable labeling f
• ? with Gibbs energy function.

Color term
Contrast term
17
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Extracting moving objects
• Unary term (or color term)

Color differences between image and background
Foreground energy
Background energy
18
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Using Background Cut (ECCV 2006)
• Using min-cut to get a smooth segmentation of
  foreground objects.
• Only considering 2D image (not video).

19
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Extracting moving objects
• Binary term (or contrast term)

In case of same labeling, energy is decreased. In
case of different labeling, energy defends on
differences between neighboring intensities.
20
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Using min-cut algorithm
• - Moving objects are extracted.

21
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Extracting moving objects
• Constructing a mask of all foreground pixels in
  space-time volume.
• Applying a 3D morphological dilation on mask.
• Finally we can obtain activity tubes (or
  objects).

t
x
Example of tubes
22
Webcam-Synopsis Peeking Around the World

• Moving Objects(tubes) Extraction
• Object queue
• Obtained activity tubes are saved in queue.

End of online phase
23
Webcam-Synopsis Peeking Around the World

• RESPONSE PHASE

24
Webcam-Synopsis Peeking Around the World

• Response phase
• User query
• I would like to watch in one minute a synopsis
  of the video from this camera captured during the
  last hour.
• I would like to watch in five minutes a synopsis
  of the last week.

When user query occurred, response phase are
started.
Input video
Synopsis
25
Webcam-Synopsis Peeking Around the World

• Response phase
• User query
• I would like to watch in one minute a synopsis
  of the video from this camera captured during the
  last hour.
• I would like to watch in five minutes a synopsis
  of the last week.

26
Webcam-Synopsis Peeking Around the World

• Creating time lapse background video
• Time lapse background video (Bout)
• The background of the synopsis video
• It should represent the background changes over
  time.
• Day-night transitions, etc.
• It should represent the background of the
  activity tubes.

Time lapse BG video
Background Video
Sampling uniformly
27
Webcam-Synopsis Peeking Around the World

• Creating time lapse background video
• Time lapse background video
• A temporal activity histogram Ha
• A uniform temporal histogram Ht

28
Webcam-Synopsis Peeking Around the World

• Creating time lapse background video
• Time lapse background video
• A temporal activity histogram Ha
• A uniform temporal histogram Ht
• Interpolating the two histograms
• Hi Ha (1- ?)Ht

Time lapse BG video
Background Video (b)
Sampling with Hi
Bout
29
Webcam-Synopsis Peeking Around the World

• Selecting tubes and stitching
• Definition

temporal mapping (or time shift)
tube with time segment
shifted tube with time segment
Input video
Synopsis
30
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Energy for temporal mapping M

Queue
target tube
other tube
user selected weights
31
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Graph

Node number number of tube
Longest tube
tube
tube
Synopsis
tube
t
tube
label
label
Label number of frame t
label
label
32
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Energy for temporal mapping M
• Unary term
• Activity Cost

Synopsis
Only pixel that were not entered into the
synopsis are added to the activity cost.
33
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Energy for temporal mapping M
• Unary term
• Consistency with background

Synopsis
border of the mapped activity tube
34
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Energy for temporal mapping M
• Binary term
• Collision Cost

Synopsis
This expression give a low penalty to pixel whose
color is similar to the background
35
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• Energy for temporal mapping M
• Binary term
• Temporal Consistency Cost

Synopsis
Preserving the chronological order of events
36
Webcam-Synopsis Peeking Around the World

• Select tubes and stitching
• E(M) is minimized by Min-cut algorithm.
• Stitching the synopsis video
• To make more reliable result
• ? The aPoisson Image Blending

End of response phase
37
Webcam-Synopsis Peeking Around the World

• Conclusion
• The method to creating a short video that is a
  synopsis of and video stream has been presented.
• Discussion
• Contribution
• Found out new interesting application.
• Can we find more contribution?
• Finding higher dimensional problem and Solving
  it.
• 3D volume animation, etc.