Multimodal Registration of Medical Data

1
Multimodal Registration of Medical Data

• Prof. Leo Joskowicz
• School of Computer Science and Engineering
• The Hebrew University of Jerusalem

2
Outline of the talk

• Introduction
• Classification of registration methods
• Rigid registration methods
• Deformable registration methods
• Challenges and research directions

This is only an introductory overview!
3
What is medical multimodal registration?
The process of establishing a common geometric
reference frame between two or more data sets
from different modalities taken at different
times for the purpose of improving preoperative
and intraoperative information for diagnosis and
navigation
4
Multimodal registration
Preoperative
Intraoperative
X-rays
CT
MRI
Fluoro
Tracking
Open MR
US
NMR
CAD
US
Special sensors
Video
Registration
Combined Data
5
Why multimodal integration?

• Combine different types of information CT/MRI,
  MRI/NMR, ...
• Track relative position of instruments and
  anatomy during surgery CT or MRI/tracker.
• Compare before and during information
  MRI/Ultrasound, CT/Xray, ...
• Supplement the quality/field of view of
  preoperative info with intraoperative info
• Clinical applications usually require more than
  one registration registration chains.

6
Registration of MRI and NMR
Ref_MRI
Ref_NMR
7
Registration Ultrasound and Doppler images
8
Registration of preoperative CT and
intraoperative tracker data
9
2D/3D X-ray/CT registration
preoperative CT slices
intraoperative X-ray images
10
Registration chain
Before surgery
During surgery
optical tracker
instruments
fluoroscopic images
3D surface model
patient
11
Not one but many registration problems!

• Many two, three, and n-way multimodal integration
  problems!
• Great differences depending on
• the type of data to be matched
• the anatomy that is being imaged
• the specific clinical requirements of procedures
• Accuracy, assumptions, and technical requirements
  vary greatly from type to type!

12
Generic registration problem
data set 2
data set 1
13
Generic registration procedure

• 1. Distortion correction and camera calibration
  for each modality
• while dissimilarity gt 0 and improvement do
• 2. Feature extraction
• 3. Feature pairing
• 4. Similarity formulation and outlier removal
• 5. Dissimilarity reduction (optimization)

Great differences in each step depending
on images and task!
14
Generic registration problem
data set 2
data set 1
Similarity formulation
Dissimilarity reduction
15
Classification of registration methods

• Dimensionality
• Type of registration basis
• Nature and domain of the transformation
• Interaction
• Optimization procedure
• Modalities
• Subject and anatomy

16
Dimensionality

• Spatial
• 2D/2D slices of MRI, CT, NMR, portal images
• 2D/3D Xray/CT, US/CT, video/CT
• 3D/3D MRI/CT, NMR/MRI,
• Temporal
• slow comparison of data sets, e.g., bone growth
• fast beating heart, angiography, injected
  imaging agents

17
Registration basis

• Image extrinsic objects attached to the patient
• invasive stereotactic frame, fiducials (screws)
• non-invasive frame, dental adapter, skin
  fiducials
• Image intrinsic image content only
• landmark based anatomical or geometric
• segmentation based rigid or deformable models
• voxel based reduction (scalars, vectors), image
  contents
• Non-image data from other sources
• trackers, laser scanners, robot arms

18
Registration transformation
19
Interaction

• Interactive (manual)
• initialization supplied
• no initialization supplied
• Semi-automatic
• user initialization
• user steering/correcting
• both
• Automatic

20
Optimization procedure

• Parameters computed
• Parameters searched for
• Mathematical characteristics
• optimization function linear, nonlinear
• solution method SVD, Lavenberg-Marquard
• Multistep approach
• fast but approximate for coarse registration,
  followed more expensive but more precise for fine
  registration

21
Modalities

• Monomodal
• CT, MR, PET, Xray, US, video, portal
• Multimodal
• CT/MR, CT/NMR, MR/NMR
• Xray/CT, video/CT
• Modality to model
• model can be atlas, CAD model, etc.
• Patient to modality
• tracker data, robot arm, etc.

22
Subject and anatomy

• Subject
• intrasubject, intersubject, atlas
• Anatomy
• head brain and skull, eye, maxillofacial
• thorax entire, cardiac, breast
• abdomen general, kidney, liver
• pelvis and perineum
• limbs femur and tibia, humerus, hand
• spine and vertebra

23
Rigid registration

• Rigid transformation
• Applicable to rigid structures which change their
  position but not their shape
• bones of the same patient
• implanted fiducials, stereotactic frames
• approximation for quasi rigid structures (brain)
• as a first step to deformable registration
• Widely used in
• orthopaedic aplications
• data from CT, Xray, trackers

24
Deformable registration

• General curved mapping
• Necessary for matching soft tissue organs and for
  cross-patient comparisons
• brain images before and during surgery
• anatomical structures at different times or from
  patients tumor growth, heart beating, compare
• matching to atlases
• Much more difficult than rigid registration!
• problem is ill-posed solution is not unique
• error measurements and comparisons are difficult

25
Rigid registration techniquestechnical
classification

• Two main approaches
• Geometric approach use spatial disparity between
  selected features to reduce difference
• distance between two matching points
• Intensity-based approach use the pixel intensity
  values to reduce difference
• intensity gradient between two pixels or voxels
• mutual information maximize image correlation

26
General rigid registration problemsgeometric
approach

• 3D/3D point to point registration based on least
  squares minimization
• 2D/3D line to point registration
• Iterative Closest Point (ICP) algorithm
  automatic feature paring
• Octree splines hierarchical data representation
• actual use landmarks cloud of points

27
Rigid registration basic concepts

• Features points, lines, surfaces
• Feature pairings predefined or automatic
• point/point, point/line, spline/spline
• Similarity measure sum of distances between
  pairwise features
• Dissimilarity reduction minimize sum of distances

28
Rigid registration mathematics (1)

• Attach coordinate systems to each data set S1, S2
• Define the rigid transformation P from one data
  set to the other.
• Transformation rotation and translation
• Goal for all points in
  data sets

29
Rigid registration mathematics (2)

• Rotation matrices
• Euler angles
• Quaternions

30
Registration mathematics (3)

• n pairs of points (pi, qi)
• Distance between pairwise points
• Difference metric sum of pairwise distances
• Dissimilarity reduction minimize sum of paiwise
  distances

31
Registration mathematics (4)

• Solving the minimization problem
• closed-form solution for three points
• closed form solution of min problem (Horn)
• nonlinear optimization methods Powell,
  Lavenberg-Marquard (numerical recipes in C)
• quadratic optimization (NNLS) of approximation
• for small angle vectors ?
• Robust estimation establish threshold for
  distance between pairs and eliminate those with
  distance higher than threshold.

32
Three points closed form solution

• Match three points in two coordinate systems left
  pL1, pL2 , pL3
• right pR1, pR2, pR3
• Choose p1 to be the origin.
• Construct x axis
• Construct y axis

33
Three points closed form solution(2)

• Construct z axis z x x y
• Build rotation matrices for two points sets
• RLxL, yL, zL and RRxR, yR, zR
• The rotation between right and left is
• RRLRRT
• The translation is tpL1 R(pR1)

34
Three points closed form solution(3)

• Problems with this solution
• This method does not use the information about
  each of the three points equally
• It cannot use the information of more than three
  points when available.
• Numerical stability problems.

35
Horn closed form solution (1)

• Given n points in two coordinate systems
• right pRi and left pLi.
• Error for each point ei pRi R(pLi) t
• Find R and t that minimize the sum of squared
  errors
• Translate all points to their centroids

36
Horn closed form solution (2)

• New error term ei pRi R(pLi) t,
• t t cR R(cL)
• The sum of the square errors
• Middle sum expression equals to 0
• Last sum expression is minimized when
• t cR R(cL) (desired translation)

37
Horn closed form solution(3)

• We should minimize then (for R)
• First and last terms are constants independent of
  R
• We should maximize the second term
• We represent R using unit quaternions, so we get

38
Horn closed form solution(4)

• Using quaternion properties, the expression can
  be written as
• Quaternion products can be expressed using
  matrices

39
Horn closed form solution(5)

• From the sum, we get
• qTNq.
• The vector q which maximizes qTNq is the
  eigenvector corresponding to the most positive
  eigenvalue of the matrix N.
• Define

40
Horn closed form solution(6)

• We can express N as
• And we return the rotation matrix R represented
  by the unit quaternion q, and the translation
  vector t, calculated after we have R.

41
Horn closed form solution(7)

• Advantages of the Horn closed-form solution
• Best possible solution is achieved by one step
  without iteration.
• No need for initial good guess to bring us close.
• All the information in the data sets is used.
• Symmetry of solution (it gives the exact inverse
  of the best transformation in the other
  direction).

42
Iterative closest point algorithm (1)Besel and
McKay, 1992

• The main problem which features to pair?
• Heuristic
• pick a set of predefined features in one data set
  
• choose the closest feature to each in the other
  data set.
• solve the problem, bringing the data sets closer
• repeat the pairing selectiob until the distance
  is minimized.

43
Iterative closest point algorithm (2)

• ICP always converges monotonically to a local
  minimum with respect to the mean-square distance
  objective function.
• Works when the data sets are reasonably close
  --gt requires a good initial guess.
• Closest point operation is the most expensive
  operation --gt data structure for fast access
  (octrees, see later).

44
Rigid registration examples

• 3D/3D cloud of points to cloud of points
• CT/CT, laser scanner/CT
• 3D/3D ridge lines to ridge lines
• CT/CT
• 3D/3D tracker cloud of points to CT points
• registration for intraoperative navigation
• 2D/3D contour lines to CT points
• anatomical image-based registration
• Octree splines

45
3D/3D countour point registration
points from 3D contour
points from CAD model
46
3D/3D tracking points to CT data
cloud of points from tracker
points from CT
47
3D/3D ridge lines registration
Advantage very few features to match!
48
2D/3D line/point registration
49
Octree spline subdivision

• Hierarchical space subdivision
• Reduces query time from
• O(n) to O(log n)

Example of quadtree subdivision
50
2D/3D registration of simulated image and femur
octree
51
3d/3d registration with octree vertebra