Mesh Modeling, Reconstruction and Spatio-Temporal Processing of Medical Images

1
Mesh Modeling, Reconstruction and
Spatio-Temporal Processing of Medical Images

• Jovan G. Brankov

Supported by the National Institutes of Health
under grants HL65245 and Whitaker Foundation.
2
Nuclear medicine SPECT

• Measuring the concentration of injected
  radioisotope bounded to the substance of interest

GE Medical Millennium Hawkeye
3
Nuclear medicine

• Myocardium perfusion
• myocardium ability to obtain substance from blood

gMCAT D1.01 University of Massachusetts Medical
School, Worcester, MA
4
Nuclear medicine

• Analytic reconstruction (no smoothing)
• Can you spot the myocardium perfusion defect?

5
Signal Processing

• Spatial smoothing

6
Signal Processing

• Spatio-temporal smoothing (no motion compensation)

7
Signal Processing

• Spatio-temporal smoothing with motion
  compensation

8
Signal Processing

• Tracking tissue elements
• Non pixel representation

9
Project overview

• Combat the noise by signal processing
• Project objective
• Motion compensated 4D reconstruction method for
  cardiac perfusion reconstruction
• (3D space 1D time)
• Proposed approach
• Content-Adaptive Mesh Modeling

10
Content-adaptive mesh modeling

• Non-uniform sampling replaces conventional pixels
• Put more samples (nodes) in high-frequency image
  regions
• Partition image domain into non-overlapping
  patches, called mesh elements
• Image function interpolated over each element
  from its nodal values

mesh node interpolation basis function
N number of mesh nodes
Mesh element
11
Potential advantages

• Mesh modeling is a compression algorithm, thus
• reduce number of unknown to be estimated
• shorter computation time
• lower memory requirement
• Built-in spatially-adaptive smoothing
• Natural framework for 4D reconstruction with
  motion tracking (deformable mesh) (our goal)

12
Intermediate results

• Many useful developments occurred along the way
• New mesh-generation techniques (2D, 3D,
  Vector-valued)
• A Fast algorithm for Accurate Content-Adaptive
  Mesh Generation, Submitted to IEEE Tran. Image
  Processing.
• A Fast algortiham for Accurate Content-Adaptive
  Mesh Generation, IEEE ICIP01
• Content Adaptive Mesh Modeling for Tomographic
  Image Reconstruction, IEEE ICIP01
• Content-Adaptive 3D Mesh Modeling for
  Representation of Volumetric Images, IEEE ICIP02
• New mesh-based reconstruction method (useful
  even for static images)
• Tomographic Image Reconstruction Based on
  Content-Adaptive Mesh Model, Submitted to IEEE
  Medical Imaging
• Tomographic Image Reconstruction using
  Content-Adaptive Mesh Modeling, IEEE ICIP01
• Content-Adaptive Mesh Modeling for Fully-3D
  Tomographic Image Reconstruction, IEEE ICIP02
• Dual-modality imiging
• Multy-Modality Tomographic Image Reconstruction
  using Mesh Modeling, IEEE ISBI'02
• 4D methods
• Motion-compensated 4D processing of Gatet SPECT
  Perfusion Studies, IEEE MIC02
• 4D Processing of Gated SPECT Images Using
  Deformable Mesh Modeling, Fully 3D Image Recon.
  2001
• Spatio-temporal clustering algorithms
• Image-sequence Segmentation based on an EM
  Algoritham for Similarity Clustering, IEEE MIC02
• Similarity based Clustering using The Expectation
  Maximization Algorithm, IEEE ICIP02
• Watermarking techniques

13
Outline of specific research questions

• Part I
• How to generate a content-adaptive mesh?
• Scalar function
• 2D support region
• 3D support region
• Vector valued function
• 2D support region
• Part II
• Given a mesh, how to reconstruct images?
• Tomographic image reconstruction
• 2D support region
• 3D support region
• Dual-modality

14
Outline of specific research questions

• Part III
• How to make and use a deformable content-adaptive
  mesh?
• 4D tomographic image processing
• 4D tomographic image reconstruction
• Extensions
• Image sequence processing
• Similarity clustering analysis
• Mesh modeling
• Watermarking
• Denoising

15
Part I Mesh generation
16
Fast mesh-generation method

• Start with an image (16k pixels)

• Find areas of rapid intensity variation
• (Maximum of 2nd derivative)

• Put nodes (3300 nodes) more densely in these
  area (Haftoning by error diffusion)

• Connect the dots
• (Delaunay triangulation)

• Interpolate to return to pixel representation

17
Experimental results

• Proposed method outperforms methods that
  explicitly seek to minimize interpolation error

Application to medical images (cardiac SPECT)
Original
Mesh structure
LS fit PSNR 40.57dB
18
Part I Volumetric Mesh generation
19
Mesh generation

• Based on similar error bound as for 2D Mesh
  generation
• Extract feature map (second directional
  derivative)
• Applied modified error diffusion
• The diffusion coefficient is chosen to be
  inversely proportional to its distance to the
  current voxel and directly proportional to the
  function 2nd gradient in the direction of
  diffusion.
• 3D Delaunay triangulation
• The National Science and Technology Research
  Center for Computation and Visualization of
  Geometric Structures - University of Minnesota

20
Mesh nodal position placement

Octtree method
Proposed method
21
Inside of myocardium
Proposed method
Octtree method
22
LS fitted images
Octtree method
Proposed method
23
Part I Vector Valued Mesh Generation
24
Basic idea

• Generate a mesh structure that will accurately
  represent all image plates (RGB).

vector valued mesh generation
B
G
R
extract luminance
mesh generation
25
Example image Mandrill
Quadtree from BW image
Proposed method
Original
CAMM from BW image
CAMM from BW image
Proposed method
Quadtree from BW image
Mesh structure
26
Outline

• Part I
• How to generate a content-adaptive mesh?
• Part II
• Given a mesh, how to reconstruct images?
• Tomographic image reconstruction
• 2D support region
• 3D support region
• Dual modality
• Part III
• How to make and use a deformable content-adaptive
  mesh?
• Extensions

27
Part II 2D Tomographic reconstruction
28
Mesh-based image reconstruction
Pixel basis
Mesh basis
Mesh and pixel reconstruction are essentially the
same, except region of node influence is
spatially-varying
29
Mesh-based image reconstruction
Mesh model equation
Mash based imaging equation
model errors
image (pixel domain)
image (mesh domain)
image (mesh domain)
data
system matrix
interpolation matrix

• usually and are negligible compare to
  imaging noise
• same form as pixel based imaging equation
• existing algorithms (e.g. EM, OSL-MAP,
  RBI-MAP,OSEM..) can be used
• dim(n) ltlt dim( f ).

30
Proposed mesh generation method
Step 3 Feature Map
Step 2 Apply lowpass filter (cutoff0.3p)
Step 1 FBP of projections
Step 5 Delaunay triangulation
Step 4 Nodal positions - Halftoning
Step 6 Interpolation
31
How many nodes should you use?

• Minimum description length (MDL)
• Function has broad minimum, so you can find an
  acceptable solution quickly.

where
,  
J. Rissanen, Modeling by shortest data
description, Automatica, vol. 14, pp. 465-471,
1978.
32
MDL selection of number of nodes

• At a wide range of resolutions, MDL says to use
  around 600-1000 mesh nodes, instead 4096 pixels

15mm FWHM
10mm FWHM
Optimum for 5-10mm
7mm FWHM
GE scanners resolution 6 - 10 mm FWHM _at_ 10 cm
distance
33
Evaluation by Channelized Hotelling observer
(CHO)1-3

• Bandpass filters 4 bands, 4 orientations
  (receptive field models)
• Feed 16 filter outputs to Hotelling detector
• (generalized likelihood ratio detector) to make
  binary decision
• - lesion present / lesion absent
• Show simulated images to numerical observer
• Best algorithm / parameter values are ones that
  cause observer to give best detection accuracy
  (area under ROC curve, Az)

Frequency domain
Space domain
1. K.J. Myers and H.H. Barrett, JOSA A, Dec.
1987 2. H.C. Gifford, R.G. Wells, and M.A. King,
IEEE Trans Nucl Sci, Aug. 1999. 3. S.D.
Wollenweber, B.M.W. Tsui, D.S. Lalush, E.C. Frey,
K.J. LaCroix, and G.T. Gullberg, Dec. 1999.
34
Reconstruction methods compared

• Pixel-based
• FBP
• ML-EM
• OSEM
• RBI-ML1
• OSL-MAP
• RBI-MAP2
• Mesh-based
• ML-EM
• OSEM
• OSL-MAP

1. C. L. Byrne, IEEE Trans. Image Proc., vol. 5,
pp. 792-794, 1996.2. D. S. Lalush and B. M. W.
Tsui, Phys. Med. Biol., vol. 43, pp. 875-886,
1998.
35
Detectability Area under ROC curve Az
Az vs. Number of nodes
Az vs. iteration
Best points
36
Result indicated by numerical observer
Less mesh nodes
Pixel-EM
1365
819
515
30
Perfusion defect
10
Best choice according to CHO and MDL
Less iterations
6
4
Perfusion defect
2
37
Conclusion

• Image quality
• According to CHO, mesh images are better than
  other methods
• A safer statement is that mesh images are at
  least as good as other methods
• Computational advantage
• After a few iterations, Mesh-OSEM is faster than
  OSEM
• Advantage becomes factor of 2 at 32 iterations
  for 2D
• Mesh-EM lends itself to 4D reconstruction

38
Part II 3D Tomographic reconstruction
39
3D Mesh Based Reconstruction
Original
Proposed 3D mesh PSNR 22.22dB
Pixel ML PSNR 21.46dB
FBP PSNR 17.69 dB
40
Part II Dual-Modality Reconstruction
41
Dual-modality imaging
X-ray source
Gamma cameras
X-ray detectors
GE Medical Millennium Hawkeye
42
Non-uniform sampling philosophy
University Southern California

• Use fine sampling near anatomical boundaries to
  allow (but not force) functional images to have
  boundaries there as well
• Note that nodal distance contain information
  about boundaries

43
Mesh structure generation

• Anatomical sampling

• Functional sampling

• Fusion

• Connect the dots (Delaunay triangulation)

• Mesh structure with 8887 nodes
• Non-stationary smoothing in Euclidean space
• Nodal distance reflects boundary

44
Reconstruction methods compared

• Pixel-based
• FBP
• Pixel ML
• Pixel MAP
• Mesh-based
• Mesh ML
• Mesh MAP
• Mesh MAP-W
• (If node as close smooth less)

45
Results for visual comparison
Original FBP Mesh MAP-W
Pixel MAP Mesh MAP
46
Conclusion

• Image quality
• Visual and quantitative evaluation (bias-variance
  (not shown)), suggest that mesh images are better
  than other methods

• Our aim
• Gated dual-modality motion compensated myocardium
  perfusion image sequence reconstruction

Northwestern University Medical School Gated MRI
47
Outline

• Part I
• How to generate a content-adaptive mesh?
• Part II
• Given a mesh, how to reconstruct images?
• Part III
• How to generate and use a deformable
  content-adaptive mesh?
• 4D tomographic image processing
• 4D tomographic image reconstruction
• Extensions

48
Part III 4D Processing
49
4D MC Smoothing Mesh Deformation Objective
Function

• Displace the mesh nodes so that the corresponding
  mesh elements in two frames achieve the best
  match in terms of their image values

first term is the matching error Ed is a
measure of mesh regularity Wm trade-off
constant between mesh matching accuracy and mesh
regularity.
Y. Wang, O. Lee, and A. Vetro, Use of
two-dimensional deformable mesh structures for
video coding. II. The analysis problem and a
region-based coder employing an active mesh
representation, IEEE Trans. Circuits Syst. Video
Tech., vol. 6, pp. 647 -659, 1996
50
4D MC Smoothing Results

• Other methods considered
• Same spatio-temporal filters as in proposed
  method, but without motion compensation (ST-NM)
• Spatial-only filtering.
• Results
• ST-NM method suffers from significant motion
  distortion
• Proposed (ST-DM) effectively reduces the noise,
  but maintains motion information.

51
New Mesh Deformation Objective Function

• Displace the mesh nodes so that the corresponding
  mesh elements in two frames achieve the best
  match in terms of their image values adjusted for
  brightening due to the partial volume effect (PVE)

first term is the matching error
accumulated Ed is a measure of mesh
regularity Wm trade-off constant between mesh
matching accuracy and mesh regularity.
52
Deformable mesh Visual methods comparison
Intensity matching
Adjusted intensity matching
53
4D MC Smoothing Results II
PVEMC_ST
Original

• Other methods considered
• Same spatio-temporal filters as in proposed
  method, but without PVE compensation (MC_ST)
• Same spatio-temporal filters as in proposed
  method, but without motion compensation (ST)

MC_ST
ST
54
Part III 4D Reconstruction
55
4D MC Reconstruction

• Mesh modeling for an image sequence
  representation
• Reconstruction algorithm
• 1) One E-M step
• 2) PVE - temporal filtering along the motion
  trajectories
• 3) Return to step 1).

g parameter used to control the degree of
temporal smoothing C normalization constant
56
Results III Time activity curves
57
Outline

• Part I
• How to generate a content-adaptive mesh?
• Part II
• Given a mesh, how to reconstruct images?
• Part III
• How to generate and reconstruction by a
  deformable content-adaptive mesh?
• Extensions
• Image sequence processing
• Similarity clustering analysis
• Mesh modeling
• Watermarking
• Denoising

58
Extension Similarity component analysis
59
Motivation Identifying region with similar
temporal behavior
Time activity curves (TAC)
Realistic MRI voxel-based numerical brain phantom
developed by Zubal et al.
11C Carfentanil Study JJ Frost et al.1990
I. G. Zubal, C. R. Harrell, E. O. Smith, Z.
Rattner, G. R. Ginde, and P. B. Hoffer,
Computerized three-dimensional segmented human
anatomy, Med. Phys, vol. 21, pp. 299-302, 1994.
60
Similarity component analysis (SCA)

• Proposing a new method to determine distinct TACs
  existing in an image sequence (want to neglect
  multiplicative scale factors)
• Traditional clustering algorithms are dependent
  on the signal amplitude
• Gaussian mixture models (GMM)
• (special case probabilistic PCA)
• k-means
• winner-take-all variant of GMM
• Principal component analysis (PCA)
• basis functions are orthogonal
• Independent component analysis (ICA)
• components are independent
• Clustered component analysis (CCA)1 Bouman et al.
  partially avoids the amplitude dependency
• ( also a special case probabilistic PCA)

compared with later
1C. A. Bouman, S. Chen, and M. J. Lowe,
Clustered Component Analysis for fMRI Signals
estimation and Classification, IEEE Tran. Image
Proc., vol. 1, pp. 609-612, 2000.
61
SCA Results

• Assume three classes

• Results demonstrates the feasibility of the
  proposed SCA concept.
• Among the tested methods, the proposed algorithms
  have the best accuracy and lowest computational
  complexity.

Percentage of correct classification
62
Extension Digital watermarking
63
Watermarking
Original

• Watermarking involves embedding a packet of
  additional digital data directly into the image.
• Main requirements for watermarking are
• no effect on the image quality
• detectable in image subject to distortions
• digital distribution (e.g. compression
  algorithms)
• image manipulation (rotation, translation,
  distortion).
• Usually compression does not affect the watermark
  significantly.
• A small geometric distortion, such as rotation,
  scaling, translation, shearing, random bending or
  change of aspect ratio can defeat most of the
  existing watermarking schemes.
• In this work
• A modification on code division multiple access
  (CDMA) watermarking method for embeds a multi-bit
  signature in the discrete cosine transform (DCT)
  domain was used.

Watermarked
64
Proposed method
Rectangular grid
We propose the use of deformable mesh model for
geometric bending correction
Grid after attack by random bending
65
Attacked image
Watermarked
Attacked
Difference
66
Corrected image
Original
Corrected
Mesh structure
Difference
67
Results Bit error rate (BER)
Wang, Y. and Lee, O., "Active mesh-a feature
seeking and tracking image sequence
representation scheme," IEEE Trans. Image Proc.,
vol. 3, pp. 610-624, 1994.
68
Conclusion

• We presented developments of
• New mesh-generation techniques (2D, 3D,
  Vector-valued)
• New mesh-based reconstruction method (useful
  even for static images)
• Dual-modality imaging
• 4D methods
• Spatio-temporal clustering algorithms
• Watermarking techniques
• Future work
• Clinical evaluation
• Further development of fully 4D methods
• Join motion and image estimation
• Use SCA for clinical task

69
Acknowledgments

• Dr. Miles WernickDr. Yongyi Yang
• Dr. Nikolas GalatsanosDr. Henry Stark
• Dr. Dean Chapmen
• Dr. R. M. Leahy University of Southern
  California
• Dr. Geoffrey Williamson