Image Fusion for

1

• Image Fusion for
• Context Enhancement
• and Video Surrealism

Adrian Ilie UNC Chapel Hill
Ramesh Raskar Mitsubishi Electric Research
Labs, (MERL)
Jingyi Yu MIT
2
(No Transcript)
3
Dark Bldgs
Reflections on bldgs
Unknown shapes
4
Well-lit Bldgs
Reflections in bldgs windows
Tree, Street shapes
5
Night Image
Background is captured from day-time scene using
the same fixed camera
Context Enhanced Image
Day Image
6
Mask is automatically computed from scene
contrast
7
But, Simple Pixel Blending Creates Ugly
Artifacts
8
Pixel Blending
9
Pixel Blending
Our MethodIntegration of blended Gradients
10
Outline

• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism

11
Gradient field
Nighttime image
x
Y
G1
G1
I1
Mixed gradient field
x
Y
G
G
Importance image W
I2
x
Y
G2
G2
Final result
Daytime image
Gradient field
12
Reconstruction from Gradient Field

• Problem minimize error Ñ I G
• Estimate I so that
• G Ñ I
• Poisson equation
• Ñ 2 I div G
• Full multigrid
• solver

GX
I
GY
13
Why Gradient-based Approach

• Comparison of intensity values are important
• Maintain gradients to capture local variations
• Directly solve for desired gradients
• Maintain subtle details
• Mix dissimilar images
• No need for precise segmentation

14
Comparison

• Average
• Subtle details are lost
• Pixel-wise blending
• Sharp transitions

15
Issues

• Boundary conditions
• Color shifts

16
Boundary Conditions

• Assumed Neumann condition at borders,
• Ñ I N 0,
• Enforced by haloing image with blacks

17
Color Shift

• Mixing dissimilar images
• Goal final image appearance matches input images
  at corresponding pixels
• Ifinal(x,y) c1 Ipoisson(x,y) c2
• Solve
• ?Wi(x,y) Ioriginal(x,y) c1 Ipoisson(x,y) c2
• Each color channel reconstructed separately

18
(No Transcript)
19
Outline

• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism

20
(No Transcript)
21
(No Transcript)
22
Overview of Process
Day time image By averaging 5 seconds of day
video
Original night time traffic camera 320x240 video
Input
Output
Enhanced video Note exit ramp, lane dividers,
buildings not visible in original night video,
but clearly seen here.
Mask frame (for frame above) Encodes pixel with
intensity change
23
Algorithm
Frame N
Gradient field
Mixed gradient field
TimeAveraged importance mask
Processed binary mask
Final result
Gradient field
Daytime image
Frame N-1
24
Outline

• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Related Work
• Surrealism

25
Related Work

• Spatio-temporal Composition
• Duchamp (Nude descending a staircase)
• Freeman 2002
• Fels 1999, Klein 2002, Cohen 2003
• Gradient-based Techniques
• Multi-spectral Socolinsky 1999
• Shadow removal Weiss 2001
• High dynamic range Fattal 2002
• Image editing Perez 2003
• Some at Siggraph04

26
Surrealism
Rene Magritte, Empire of the Light
27
Outline

• Context Enhancement
• Gradient-based Fusion
• Video Enhancement
• Surrealism

28
Time-lapse Mosaics
Maggrite Stripes
time
29
Time Lapse Mosaic
30
Time Lapse Mosaic
31
t
32
Sunrise at Night
33
BiSolar System
34
Discussion

• User Experience
• More effective in conveying scene context
• Dreamy appearance
• Nonrealistic False conditions
• Applications
• Tools for artists
• Surveillance
• Amusement park rides
• Performance
• 1 sec/frame for 320x240
• 3 min for 4Mpixel image

35
Image Fusion for Context Enhancement

• Nonrealistic but comprehensible context
• Fusion using multiple images
• Enhancing night images with day bgrnd
• Gradient-based fusion
• Video surrealism tools

t