Video Summarization of Key Events Stage I - The Critical View

1
Video Summarization of Key EventsStage I - The
Critical View

• Michael A. Grasso, MD, PhD
• University of Maryland School of Medicine
• UMBC Computer Science
• MichaelGrasso.com

2
Abstract

• Laparoscopic surgery is a minimally invasive
  technique with unique training requirements.
  Video-assisted evaluation is one method that
  surgical residents can use to demonstrate
  competence. Automated video summarization can
  increase the efficiency of evaluations by
  directing the senior surgeon to key portions of a
  surgical procedure. We are using image
  classification techniques to segment videos of
  laparoscopic cholecystectomies to assist with
  surgical training and evaluation.

3
Overview

• Background
• Laparoscopic Surgery
• Image Classification
• Methods
• Discussion

4
Laparoscopic Surgery

• Minimally Invasive Surgery.
• First performed in 1987.
• Used in many surgical procedures.
• Gall bladder removal (cholecystectomy).
• Esophageal surgery (fundoplication).
• Colon surgery (colectomy).
• Others.

5
Laparoscopic Approach

• Narrow tubes (trocars) are inserted into the
  abdomen through small incisions.

www.fda.gov
6
Laparoscopic Procedure

• Camera is passed
• through trocar.
• Procedure is often
• videotaped.
• Carbon dioxide is
• infused through
• trocar.
• Instruments are passed through the trocars to
  cut, manipulate, and sew.

7
Laparoscopic Aftercare

• Compared with an open procedure.
• Smaller scars.
• Reduced pain.
• Quicker recovery.

http//www.nlm.nih.gov/medlineplus/ency/presentati
ons/100166_1.htm
8
Technical Challenges

• Access limited to small incisions.
• Long instruments with only the tips visible.
• Two-dimensional video.
• Limited tactile feedback.

British Journal of Surgery. 2004
Dec91(12)1549-1558
9
Laparoscopic Training

• Traditional apprenticeship model.
• Acquire skills during actual procedures.
• Not sufficient for laparoscopic skills.
• Other methods.
• Box trainer with animal or synthetic models.
• Virtual reality simulator.
• Video-based assessment.

10
Assessment of Skills

• Trainee must demonstrate competency.
• Evaluation by a senior surgeon.
• Direct observation of the trainee.
• Video-based assessment.
• Question Can we organize video in order to
  assist in video-based assessment?

American Journal of Surgery. 1991
Mar161(3)399-403
11
Objective

• Identity key portions of surgical procedure to
  aid in video-based assessment.
• Stage I is to identify the "critical view".

12
Overview

• Background
• Laparoscopic Surgery
• Image Classification
• Methods
• Discussion
• Summary Organize surgical video to make it
  easier for expert to review.

13
The Critical View

• Helps ensure that the anatomy has been properly
  identified.
• Occurs after dissecting anatomy.
• Occurs before clipping the cystic artery and
  cystic duct.

14
The Critical View
Fundus
Cystic artery
Liver
Cystic duct
Netter's Atlas of Human Anatomy
15
The Critical View
16
Image Classification - Human

• Features a person might use.
• Spectral features.
• Tonal variations.
• Textural features.
• Spatial distribution of tonal variations.
• Contextual features.
• Features from surrounding areas.

17
Image Classification - Computed

• Features extracted from image.
• Spectral features.
• Distribution, size, width.
• Textural features.
• Homogeneity, contrast, correlation.
• Similarity/distance metrics.
• Jaccard coefficient, Jeffrey divergence.

Journal of WSCG. 2003 11(1)269-273 IEEE
Transaction on Systems, Man, and Cybernetics.
1973 Nov 3(6)610-621
18
Color Histogram

• Red, green, blue, or gray.
• Count number of pixels for each tone.
• One 28 set for an 8-bit image for each color.
• Does not vary with translation and rotation.
• Ignores shape and texture.
• 4x4 image.
• 4 gray tones.
• H 5, 4, 5, 2

0 0 1 1
0 0 1 1
0 2 2 2
2 2 3 3
19
Binary Histogram

• Quantize values for each tone to 0 or 1.
• Background color given less weight.
• Subtle changes given more weight.
• HB 1, 0, 1, 1

0 0 0 0
0 0 0 0
0 2 2 2
2 2 3 3
20
3D Histogram

• Distribution within a 3D color-space.
• 3D color space (red, green, blue).
• Used in object recognition image retrieval.
• n3 entries, where n number of tones.
• Example.
• Quantized to 3 tones
• for each color.

21
Spatial-Dependency Matrix

• Co-occurrence matrix.
• Co-occurring values (0o, 45o, 90o, 135o).
• Four 28 x 28 matrices for 8-bit image.

135o 90o 45o
0o Ref 0o
45o 90o 135o
Co-occurring Bits Co-occurring Bits Co-occurring Bits Co-occurring Bits
0 1 2 3
Reference Bits 0 4 2 1 0
Reference Bits 1 2 4 0 0
Reference Bits 2 1 0 6 1
Reference Bits 3 0 0 1 2
0 0 1 1
0 0 1 1
0 2 2 2
2 2 3 3
M0
22
Additional Spectral Features

• Location of the distribution.
• Mean S (binfreq) / S (freq).
• Mode bin of the max freq.
• Size of the distribution.
• Standard deviation.
• Width of the distribution.
• Max(bin) - Min(bin).

23
Additional Textural Features

• Homogeneity.
• Number of tone transitions.
• Contrast.
• Amount of local variation.
• Correlation.
• Measure of linear dependencies.

24
Similarity/Distance Metrics

• Jaccard Coefficient.
• Similarity of two sample sets.
• A ? B / A ? B
• Two binary sets.
• M11 / (M01 M10 M11)
• Jeffrey Divergence.
• Distance between two vector spaces.
• S (xi log(xi/avgi) yi log(yi/avgi))

n
i1
25
Other Distance Metrics

• City Block or Manhattan Distance.
• Euclidean Distance.
• Chi-Square.
• Canberra Distance.

Proceedings ACM SAC. 20081225-1230
26
Related Efforts - Hysteroscopy

• Use Jeffrey divergence on color histogram to
  identify segments.
• Relevant segments based on image redundancy.
• No understanding
• of the content
• of each segment.

Mayo Clinic
Proceedings 27th IEEE-EMBS. 20055680-5683
27
Related Efforts - Echocardiogram

• Use cosine similarity and edge change ratio to
  identify video segments.
• State-based modeling.
• Identify states in each
• video segment.
• Diastole (resting).
• Systole (contracting).

Medline Plus
IEEE Transaction on Information Technology in
Biomedicine. 2008 May12(3)366-376
28
Overview

• Background
• Laparoscopic Surgery
• Image Classification
• Methods
• Discussion
• Summary Spectral and textural features compared
  with similarity metrics.

29
Methods

• Our objective.
• Identity key portions of surgical procedure to
  aid in video-based assessment.
• Stage I is to identify the "critical view".

30
Tools

• FFmpeg
• http//ffmpeg.mplayerhq.hu/
• Extract JPEG images.
• ImageJ
• http//rsbweb.nih.gov/ij/
• Macros and Java plugins.

31
Work Plan

• Identify videos for analysis.
• Convert videos to JPG.
• Evaluate ability to identify critical view.
• Color histogram.
• Binary histogram.
• 3D histogram.
• Spatial-dependency matrix.
• Jaccard coefficient, Jeffrey divergence.

32
Algorithm
Color Histograms Binary Histograms 3D
Histograms Spatial-Dependency Matrices
Feature Extraction ImageJ
Random Image
Image Extraction FFmpeg
Critical View
Similarity Metric
Critical View?
33
Overview

• Background
• Laparoscopic Surgery
• Image Classification
• Methods
• Discussion
• Summary Attempt to identify the critical view by
  comparing image features with similarity metrics.

34
Discussion

• Color and binary histograms do not correlate with
  the critical view.
• They do, however, predict when we are in the
  abdomen.
• Currently working on 3D histograms and
  spatial-dependency matrices.
• NIH grant application under development.

35
Challenges

• Live tissue (vs. solid objects).
• Deformable.
• Normal variation.
• Disease states.
• May need to consider.
• Temporal information.
• Relevant clinical data of the patient.
• Critical view "rectangle" (contextual).

36
Summary

• We are comparing image features with similarity
  metrics to identify the critical view.
• This is a first step in automated video
  summarization, to help with video-assisted
  evaluation of laparoscopic surgery.

37
Thank You