Automated neuroanatomical microvessel segmentation, nuclei labeling, and geometric computations with

1
Automated neuroanatomical microvessel
segmentation, nuclei labeling, and geometric
computations with 3D two photon laser scanning
microscopyJoint Work J. P. Kaufhold, P. S.
Tsai, P. Binder, B. Friedman, P. D. Lyden, D.
Kleinfeld and A.I. Ifarraguerri

• John Kaufhold, Ph.D.
• SAIC Intelligent Systems Division
• February 1st, 2008

Grant Manipulating Neural Tissue With Ultrashort
Laser Pulses NIH NIBIB BRP R01 EB003832
2
2000s
See Paul Frenchs and Jelena Kovacevics talks,
e.g.
3
The Microscope
4
Microvessel segmentation, nuclei labeling, and
geometric computations
RESULT
Tsai, Blinder, Kaufhold, Friedman, Ifarraguerri,
Lyden, Kleinfeld (in progress)
5
Related work Francis Cassot, Frederic Lauwers,
Celine Fouard, Prohaska, and Lauwers-cances
Valérie. Microcirculation, 13(1)1-18, 2006.
6
Enabling technology 1 All Optical Histology
High Fluence Ultrashort Laser Pulses for
Ablation Threshold Fluence 1J/cm2
1mJ/(3mm)2 Combined with Low Fluence Pulses for
TPLSM Imaging
7
Enabling technology 2 Multiply Labeled
Tissue (vasculature nuclei)
8
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

9
Sample control (age, weight, sex)
10
Sample control (age, weight, sex)
legend
11
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

12
Gain Correction
Before Gain Correction
Mean projection across 400 z-frames
13
Registration And Tiling
Block 1
Block 2
Maximize correlation coefficient of
intensities in overlap
Block n
14
Channel Subtraction
D DAPI channel only somata
F FITC channel mostly vessels some cell nuclei
A Alexa 594 channel labeled neuronal nuclei
some vessels some cell
nuclei
D F A
15
(No Transcript)
16
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

17
Grid volume into overlapping blocks
Registered and Tiled Volume
Block size 100x100x100 voxels
Strip data size 300x700x1000 voxels AOH data
size 1000x1000x1000 voxels
18
Restitching Binary Volumes
This process is repeated for all block overlaps,
reducing segmentation edge effects.
19
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

20
Produce Binary Somata Masks
INPUT
Raw Block
Filtering 1) regularizes somata shape and 2)
separates SM nuclei intensities from background
21
Extract Nuclear Centroids From Mask
INPUT
OUTPUT
2D illustration of 3D steps to isolate centroids
from masks
22
Validate Nuclei Centroid Estimates
Nuclei Mask Overlays
Nuclei Centroid Overlay (single slice)
Nuclei Centroid Overlay (10 m projection)
23
Nuclei Centroid Estimation Performance
automated algorithm
hand-labeled centroids
11 correspondence, Automated method to hand label
mT
False Alarms
Misses
Scoring ground truth centroids vs. automatic
centroid location estimates
24
Association distance lt 10 microns
25
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

26
Classification of Nuclei Types
Aim Separate neuronal nuclei from non-neuronal
nuclei Method Iteratively reestimate classes
average intensity in jth background mask
Nucleus to background difference is computed per
jth centroid
27
Classification of Nuclei Types
, compute decision boundary
At ith iteration,
28
Classification of Nuclei Types
Counts
Ratio v1 / mean (v2, v3, vn)
29
Classification of Nuclei Types
Counts
Ratio v1 / mean (v2, v3, vn)
30
Classification of Nuclei Types
Counts
Ratio v1 / mean (v2, v3, vn)
31
Classification of Nuclei Types
OUTPUT
Counts
Ratio v1 / mean (v2, v3, vn)
32
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

33
Vessel Segmentation Cascade
INPUT
See also Frangi, Niessen, Vincken, Viergever,
1998 Fridman, Pizer, Aylward, Bullitt, 2003
34
Vessel Segmentation Cascade
OUTPUT
Kirbas Quek Review, 2000 Viola Jones Detector
Cascade, CVPR 2001
35
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

36
Morphometry Restoring Anisotropy Corrections
Cortical Surface
R into Cortex
INPUT
Vascular Mask, VM Orthographic projection (prior
to aniostropy correction)
Observation Vessel footprints at left thinner
than footprints at right. This is due to PSF
elongation in imaging depth direction. Issue Bias
es microvascular density estimates vs. depth into
cortex and complicates capillary isolation.
Imaging Depth into plane
Imaging Depth
500u
37
Morphometry Restoring Anisotropy Corrections
Cortical Surface
R into Cortex
OUTPUT
Vascular Mask Corrected, VC Orthographic
projection (post aniostropy correction)
Result Vessel footprints at right same width on
average as those at left. Vessel locations are
mostly preserved. Assertion Vessel diameter more
useful post-correction.
Imaging Depth into plane
Imaging Depth
500u
38
Morphometry Restoring Anisotropy Corrections
METHOD impose isotropy via binary deconvolution
Imaging Depth
39
Morphometry Restoring Anisotropy Corrections
40
Morphometry Restoring Anisotropy Corrections
Amount to symetrically shave from local vessel
segment x-secs as a function of imaging depth
9

8
7
Major to Minor Axis Diff
6
5
4
3

0
100
200
300
400
500
600
700
Imaging Depth Bin
41
Morphometry Restoring Anisotropy Corrections
INPUT
Imaging Depth
42
Morphometry Restoring Anisotropy Corrections
OUTPUT Local shaving
Imaging Depth
43

• Binary Deconvolution self-calibration issues
• Vessels parallel to the sampled plane should be
  ignored for estimating major-minor axes.
  Alternatively, a calibration phantom of small
  FITC-filled tubes (2-15 m radii) would be ideal.
• Does not take into account potential increases
  in estimated vessel diameter with increased FITC
  intensity. This factor could be absorbed into the
  local correction in all directions.
• Constrained to thin to no less than 2 voxels
  horizontally.
• Vessels x-sections with 2 components on a
  horizontal strip may fragment.

44
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

45
Capillary Network Isolation
Cortical Surface
R into Cortex
INPUT Anisotropy-corrected microvascular
mask Observation Larger vessels contribute
significantly to volume fraction, but participate
significantly less in gas exchange. Aim Remove
vessels larger than capillaries.
46
Capillary Network Isolation
Cortical Surface
R into Cortex
OUTPUT An approximation to capillary-only
microvasculature isolation. METHOD Only vessel
segments whose centerline distances to the
closest non-vessel point lt 3u are kept (i.e. lt6u
lumenal diameters).
47
Capillary Network Isolation
INPUT
All vessels projected, VC
48
Capillary Network Isolation
49
Capillary Network Isolation
50
Capillary Network Isolation
51
Capillary Network Isolation
52
Capillary Network Isolation
53
Capillary Network Isolation
54
Capillary Network Isolation
55
Capillary Network Isolation
56
Capillary Network Isolation
57
Capillary Network Isolation
58
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

59
Aim Extract vectorized model of centerline points
60
Vectorization of Microvascular Networks
Surface Rendering (2.7 of previous volume)
61
Vectorization of Microvascular Networks
Surface Rendering (2.7 of previous volume)
Centerlines
Method Francis Cassot, Frederic Lauwers, Celine
Fouard, Prohaska, and Lauwers-cances Valérie.
Microcirculation, 13(1)1-18, 2006.
62
Vectorization of Microvascular Networks
63
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

64
Cortical Flattening
INPUT
METHOD Cortical surface map estimation and
voxel-column shifting
OUTPUT
Before Cortical Surface Correction
After Cortical Surface Correction
65
(No Transcript)
66
Cell Population Count andVessel Density
Statistics
Neuronal Density
All Microvessels
N10 mice
All Cell Density
Capillaries Only
Non-Neuronal Density
67
Cell Population Count andVessel Density
Statistics
Neuronal Density
All Microvessels
N10 mice
All Cell Density
Capillaries Only
Non-Neuronal Density
68
Cell Population Count andVessel Density
Statistics
Neuronal Density
All Microvessels
N10 mice
All Cell Density
Capillaries Only
Non-Neuronal Density
69
Cell Population Count andVessel Density
Statistics
70
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

71
Neuroanatomy Result(not my expertise! See
coauthor list!)
Median Cell Density (count/mm3 tissue)
72
Neuroanatomy Result(not my expertise! See
coauthor list!)
Number of cells related to capillary volume
Median Cell Density (count/mm3 tissue)
73
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Vessel segmentation short cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

74
ACTIVE LEARNING

• Labeling large datasets

75
Traditional treadmill CAD development process (12
step program)

• Take medical image data (xi are medical image
  volumes, e.g.)
• Label image (cancer/no cancer) via radiologist,
  for better diagnosis/treatment/therapy
• Realize binary cancer/no cancer labels (yi) could
  be generated automatically via segmentation
  algorithms
• Realize segmentation algorithms are untweakable
  to desired accuracy (simultaneous
  under/oversegmentation)
• Realize with enough training data (x labeled with
  y), CAD could predict next label (requires
  machine learning/segmentation algorithms)
• Label a database of medical images (x,y)
• Train a classifier mapping x?y based on
  training data, (x,y)
• Estimate detection performance and confidence via
  validation (Cross Fold Validation, e.g.)
• Realize errors in classifier can be traced to
  specific clusters/issues
• Derive new features or tweak CAD training to fix
  error clusters/issues
• Add new class to accumulate confuser classes
  (vessel/airway/chest wall, e.g., for lung CAD)
• Incremental progress until all classes are
  included
• New modality developed to deal with difficult
  issues in legacy system (design to CAD still less
  profitable than design to M.D.)

The last decade
76
Main idea in Active Learning Focus labeling
effort on most informative (most difficult)
exemplars
77

• Classifier Accuracy Depends on
• Amount of Training Data
• Selection of Informative Examples

100 Accuracy (separable classes)
Classifier Accuracy (area under ROC curve, e.g.)
100
Percent of database instances labeled (distance
to right is proportional to labeling cost, )
Y. Abramson and Y. Freund, Active learning for
visual object detection, UCSD Technical Report.
78
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Short vessel segmentation cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

79
Recent Work and Community Needs/Issues

• Recent System Architecture Work NIH NIBIB R01,
  Reducing gigabytes of voxels to metrics on tens
  of thousands of object relationships
• Image formation, dark/gain correction,
  registration, tiling, channel subtraction
• Blocking large datasets
• Segmentation of cell nuclei
• Classification of nuclei types
• Short vessel segmentation cascade
• Morphometry-restoring anisotropy corrections
• Capillary network isolation
• Vectorization of microvasculature networks
• Cell population count and vessel density
  statistics (cortical flattening)
• Neuroanatomy resultlarger animals have more
  nuclei and blood vessel fractions
• Need for machine learning to add value Labeled
  datasets
• Issues
• Expensive
• Use active learning for truly expensive labeling
  efforts
• No glory. Academically invisible. Effort/Effect
  ratio high.
• Embarrassment of riches (i.e. data)
• Great microscopy labs produce microscopes which
  produce enormous quantities of data (giant cubes
  of numbers), but require concomitant processing
• Not immediately publishable biological
  theories/experimental results

Message 1 Theres not one algorithm to segment
everything. These algorithms are domain-specific.
Message 2 1) Create labeled databases 2)
you dont have to label all your data to get most
of the learning benefit (as long as you label
wisely).
Message 3 Data explosions are going
around. Budget more image analysis into your
proposal.
80
(No Transcript)
81
Technical Focus Advance the utility of
high-fluence ultrashort pulses, plus enabling
hardware and software tools, for
discovery-based studies in cortical
neurovasodynamics
Scientific Focus Connection between vascular
topology, flow dynamics, and neuronal control