High-performance imaging using dense arrays of cameras

1
Light field microscopy
Marc Levoy, Ren Ng, Andrew Adams Matthew
Footer, Mark Horowitz
Stanford Computer Graphics Laboratory
2
Executive summary

• captures the 4D light field inside a microscope
• yields perspective flyarounds and focal stacks
  from a single snapshot, but at lower spatial
  resolution
• focal stack ? deconvolution microscopy ? volume
  data

3
Devices for recording light fields
(using geometrical optics)

• handheld camera Buehler 2001
• camera gantry Stanford 2002
• array of cameras Wilburn 2005
• plenoptic camera Ng 2005
• light field microscope (this paper)

4
Light fields at micron scales

• wave optics must be considered
• diffraction limits the spatial angular
  resolution
• most objects are no longer opaque
• each pixel is a line integral through the object
• of attenuation
• or emission
• can reconstruct 3D structure from these integrals
• tomography
• 3D deconvolution

5
Conventional versus plenoptic camera
6
Conventional versus plenoptic camera
7

8
Digital refocusing
S

• refocusing summing windows extracted from
  several microlenses

9
Example of digital refocusing
10
Refocusing portraits
11
Macrophotography
12
Digitally moving the observer
S
S

• moving the observer moving the window we
  extract from the microlenses

13
Example of moving the observer
14
Moving backward and forward
15
A light field microscope (LFM)
eyepiece
intermediate image plane
objective
specimen
16
A light field microscope (LFM)

• 40x / 0.95NA objective
• ?
• 0.26µ spot on specimen 40x 10.4µ on sensor
• ?
• 2400 spots over 25mm field
• 1252-micron microlenses
• ?
• 200 200 microlenses with12 12 spots per
  microlens

sensor
eyepiece
intermediate image plane
objective
specimen
? reduced lateral resolution on specimen
0.26µ 12 spots 3.1µ
17
A light field microscope (LFM)
sensor
18
Example light field micrograph

• orange fluorescent crayon
• mercury-arc source blue dichroic filter
• 16x / 0.5NA (dry) objective
• f/20 microlens array
• 65mm f/2.8 macro lens at 11
• Canon 20D digital camera

200µ
ordinary microscope
light field microscope
19
The geometry of the light fieldin a microscope

• microscopes make orthographic views
• translating the stage in X or Y provides no
  parallax on the specimen
• out-of-plane features dont shift position when
  they come into focus

objective lenses are telecentric
20
Panning and focusing
panning sequence
focal stack
21
Mouse embryo lung(16x / 0.5NA water immersion)
200µ
pan
focal stack
light field
22
Axial resolution(a.k.a. depth of field)

• wave term geometrical optics term
• ordinary microscope (16x/0.4NA (dry), e 0)
• with microlens array (e 125µ)
• stopped down to one pixel per microlens

? number of slices in focal stack 12
23
3D reconstruction

• confocal scanning Minsky 1957
• shape-from-focus Nayar 1990
• deconvolution microscopy Agard 1984
• 4D light field ? digital refocusing ?3D focal
  stack ? deconvolution microscopy ?3D volume
  data

24
3D deconvolution
McNally 1999
focus stack of a point in 3-space is the 3D PSF
of that imaging system

• object PSF ? focus stack
• ? object ? PSF ? ? focus stack
• ? focus stack ? ? PSF ? ? object
• spectrum contains zeros, due to missing rays
• imaging noise is amplified by division by zeros
• reduce by regularization, e.g. smoothing

25
Silkworm mouth(40x / 1.3NA oil immersion)
100µ
slice of focal stack
slice of volume
volume rendering
26
Insect legs(16x / 0.4NA dry)
200µ
27
3D reconstruction (revisited)

• 4D light field ? digital refocusing ?3D focal
  stack ? deconvolution microscopy ?3D volume
  data
• 4D light field ? tomographic reconstruction
  ?3D volume data

28
Implications of this equivalence

• light fields of minimally scattering volumes
  contain only 3D worth of information, not 4D
• the extra dimension serves to reduce noise, but
  could be re-purposed?

29
Conclusions

• captures 3D structure of microscopic objects in a
  single snapshot, and at a single instant in time

Calcium fluorescent imaging of zebrafish larvae
optic tectum during changing visual stimula
30
Conclusions

• captures 3D structure of microscopic objects in a
  single snapshot, and at a single instant in time
• but...
• sacrifices spatial resolution to obtain control
  over viewpoint and focus
• 3D reconstruction fails if specimen is too thick
  or too opaque

31
Future work

• extending the field of view by correcting
  digitally for objective aberrations

32
Future work

• extending the field of view by correcting
  digitally for objective aberrations
• microlenses in the illumination path
• ? an imaging microscope scatterometer

angular dependence of reflection from single
squid iridophore
33
http//graphics.stanford.edu/projects/lfmicroscope