Stavri Nikolov1, Tim Dixon2, John Lewis1, Nishan Canagarajah1, Dave Bull1, Tom Troscianko2, Jan Noyes2 1Centre for Communications Research, University of Bristol, UK 2Department of Experimental Psychology, University of Bristol, UK

1
Stavri Nikolov1, Tim Dixon2, John Lewis1, Nishan
Canagarajah1, Dave Bull1, Tom Troscianko2, Jan
Noyes21Centre for Communications Research,
University of Bristol, UK2Department of
Experimental Psychology, University of Bristol,
UK

• How Multi-Modality Displays
• Affect Decision Making
• NATO ARW 2006, 21 - 25 October 2006, Velingrad,
  Bulgaria

2
Overview

• Multi-Sensor Image Fusion
• Multi-Modality Fused Image/Video Displays
• Target Detection in Fused Images with Short
  Display Times (results)
• Scanpath Assessment of Fused Videos
• Multi-Modality Image Segmentation
• Summary

3
How Does Image/Video Fusion Affect Decision Making

• Experiment 1 Target Detection in Fused Images
  with Short Display Times Decision is the target
  present or not?
• Experiment 2 Target Tracking in Fused Videos (
  secondary task) Decision where to look to
  follow the target?
• Experiment 3 Image Segmentation (decomposing an
  image into meaningful regions/object) in Fused
  Images Decision which objects to segment and
  how?

4
Multi-Sensor Image Fusion
5
Multi-Sensor Image Fusion Definition

• the process by which several images coming from
  different sensors, or some of their features, are
  combined together to form a fused image
• the aim of the fusion process is to create a
  single image (or visual representation) that will
  capture most of the important and complementary
  information in the input images and will resolve
  better any uncertainties, inconsistencies or
  ambiguities.

6
Multi-Sensor Image Fusion Example
An example
Visible and IR images courtesy of Octec Ltd, UK
7
Multi-Sensor Image Fusion Applications

• Many different applications of image fusion
• remote sensing
• surveillance
• defence
• computer vision
• robotics
• medical imaging
• microscopic imaging
• art

8
Multi-Sensor Image Fusion Applications

• Image fusion is used in
• night vision systems
• binocular vision
• 3-D scene model building from multiple views
• image/photo mosaics
• digital cameras and microscopes to extend the
  effective depth of field by combining multi-focus
  images
• target detection

9
Multi-Sensor Image Fusion Different Levels

• Image fusion can be performed at different levels
  of the information representation
• signal level
• pixel level
• feature / region level
• object level
• symbolic level

10
Multi-Modality Image Displays
11
Multi-Modality Image Displays

• Adjacent (side-by-side) displays ()
• Window displays
• Fade in/out displays
• Checkerboard displays ()
• Gaze-contingent multi-modality displays ()
• Hybrid fused displays ()
• Interleaved video displays

12
Adjacent and Checkerboard Displays
Images from the Eden Project Multi-Sensor Data
Set
13
Gaze-Contingent Multi-Modal Displays
Demo of a gaze-contingent multi-modal display
(GCMMD) using aerial photographs and maps of
England (from Multimap.com).
Multi-Modality Gaze-Contingent Displays for
Image Fusion", S. G. Nikolov, M. G. Jones, I. D.
Gilchrist, D. R. Bull, C. N. Canagarajah,
Proceedings of Fusion 2002
14
Hybrid Fused Image Displays
(1.0,0.0) (0.8,0.2) (0.6,0.4)
(0.4,0.6) (0.2,0.8) (0.0,1.0)
Hybrid Fused Displays Between Pixel- and
Region-Based Image Fusion", S. G. Nikolov, J. J.
Lewis, R. J. OCallaghan, D. R. Bull and C. N.
Canagarajah, Proceedings of Fusion 2004
15
Fused Image Assessment

• The results of image fusion are
• either used for presentation to a human observer
  for easier and enhanced interpretation
• or subjected to further computer analysis or
  processing, e.g. target detection or tracking,
  with the aim of improved accuracy and more robust
  performance
• Finding an optimal fused image is a very
  difficult problem since in most cases this is
  task and application dependent.

16
Which Fused Image is Better?
Original Visible and IR UN Camp images courtesy
of TNO Human Factors
it depends what we want to do with it, i.e. the
task we have!
17
Categories of Fused Image Assessment Metrics
A B input images
FUSION
F fused image
18
Fused Image Assessment Metrics

• A number of image quality metrics have been
  proposed in the past but all require a reference
  image
• In practice an ideal fused is rarely known and is
  application and task specific
• other metrics try to estimate what information is
  transferred from the input images to the fused
  image
• two such metrics that we used in our study to
  assess the quality of the fused images are
  Piella's image quality index (IQI) 03 and
  Petrovic's edge-based QAB/F metric 00,03 (both
  of which are IFIMs)

19
Experiment 1 Target Detection in Fused Images
Decision Is the target present or not?
20
Experiment 1, Task 1 Objective Human Task
Performance

• Testing 3 fusion schemes AVR, CP DT-CWT, and 3
  JPEG2000 compression rates clean, low (.3bpp)
  and high (.2bpp).
• Using a signal detection paradigm to assess Ps
  ability to detect presence of the soldier
  (target) in briefly displayed images.

21
Task 1 Method

• Fixation point shown for 750ms, an image
  presented for 15ms, followed by an inter-stimulus
  interval of 15ms, and a mask for 250ms.

22
Experiment 1, Task 2 Subjective Image Assessment

• Show pairs of images, ask Ps to rate both out of
  5 (5 Best quality, 1 Worst quality). Images
  paired

23
Target Detection in Fused Images Main Results

• The results showed a significant effect for
  fusion but not compression in JPEG2000 images
• Subjective ratings differed for JPEG2000 images,
  whilst metric results for both JPEG (different
  study) and JPEG2000 showed similar trends

Characterisation of Image Fusion Quality Metrics
for Surveillance Applications over Bandlimited
Channels", E. F. Canga, T. D. Dixon, S. G.
Nikolov, D. R. Bull, C. N. Canagarajah, J. M.
Noyes, T. Troscianko, Proceedings of Fusion 2005
24
Experiment 2 Target Tracking in Fused Videos
Decision Where to look to follow the target?
25
Experiment 2

• Applying an eye-tracking paradigm to the fused
  image assessment process.
• Moving beyond still images assessing
  participants ability to accurately track a
  figure.
• Using footage taken recently at the Eden Project
  Biome.
• Videos of a soldier walking through thick
  foliage filmed in both visible light and IR, and
  at two natural luminance levels.
• All videos registered using our Video Fusion
  Toolbox (VFT)

26
Original Videos Used

• High Luminance (HL)
• Low Luminance (LL)

Videos from the Eden Project Multi-Sensor Data
Set
27
Fused Videos Used

• Low Luminance
• Fused Average
• Fused DWT
• Fused DT-CWT

• High Luminance
• Fused Average
• Fused DWT
• Fused DT-CWT

28
Tasks Methods

• Participants asked to visually track the solider
  as accurately as possible throughout video
  sequence.
• Tobii x50 Eye-Tracker used to record eye
  movements.
• Participants also asked to press SPACE at
  specific points in the two sequences (when
  soldier walked past features of the scene).

• 10 Ps (5m, 5f) mean age 27.1 (s.d. 6.76).
• Each shown 6 displays Viz, IR, VizIR, AVE,
  DWT, DT-CWT.
• All Ps shown each condition in 3 separate
  sessions.
• Half shown above order first, half reverse order.
  Order switched for 2nd and switch back for 3rd
  sessions.
• Eye position and reaction times recorded.

29
Accuracy Results I

• Eye position translated onto target box for each
  participant.
• Calculated an accuracy ratio, hitstotal views
  for each condition.
• Also considered Tobii accuracy coding.

30
Accuracy Results II
Videos from the Eden Project Multi-Sensor Data
Set
31
Results (High Luminance)

• Accuracy Scores revealed
• Main effect display modality (p .001).
• No main effect of session (p gt .05).
• No interaction (p gt .05).
• Post hoc tests revealed differences between Viz
  and AVE, DWT, CWT.
• IR and AVE, DWT
• RT Scores revealed
• No significant effects

Scanpath Analysis of Fused Multi-Sensor Images
with Luminance Change", T.D. Dixon, S.G. Nikolov,
J.J. Lewis, J. Li, E.F. Canga, J.M. Noyes, T.
Troscianko, D.R. Bull and C.N. Canagarajah,
Proceedings of Fusion 2006
32
Results (Low Luminance)

• Accuracy Scores revealed
• Main effect display modality (p lt .001).
• No main effect of session (p gt .05).
• No interaction (p gt .05).
• Post hoc tests revealed differences between Viz
  and IR, AVE, DWT, CWT.
• RT Scores revealed
• Main effect of fusion IR significantly closer to
  ideal timing.

33
Target Tracking in Fused Videos Conclusions I

• The current experimental results reveal two
  methods for differentiating between fusion
  schemes the use of scanpath accuracy and RTs.
• Fused videos with higher (perceived) quality do
  not necessarily lead to better tracking
  performance
• The AVE and DWT fusion methods were found to
  perform best in the 2.1_i tracking task. From a
  subjective point, the DWT appeared to create a
  sequence that was much noisier and with more
  artefacts than the CWT method.

34
Target Tracking in Fused Videos Conclusions II

• All of the fusion methods performed significantly
  better than the inputs, highlighting the
  advantages of using a fused sequence even when
  luminance levels are high.
• Results suggest that when luminance is low, any
  method of attaining additional information
  regarding the target location will significantly
  improve upon a visible light camera alone.

35
Experiment 3 Multi-Modal Image Segmentation
Decision Which objects to segment and how?
36
Multi-Modal Image Segmentation

• Multi-modal sensors
• Multi-sensor systems
• Many applications need good segmentation
• How best to segment a set of multi-modal images?
• To study how fusion affects segmentation
• Previous evaluation methods
• Subjective
• based on ground truth
• Need for objective measure of quality of
  segmentation techniques

? sets of multi-modal images
37
Joint Vs. Uni-Modal Segmentation

• Two approaches investigated
• Uni-modal segmentation
• S1 s(I1),, SN s(IN)
• Each image segmented separately
• Different segmentations for each image in the set
• Joint segmentation
• Sjoint s(I1 IN)
• All images in the set contribute a single
  segmentation
• Segmentation accounts for all features from all
  input images

38
Uni-Modal and Joint Image Segmentation
Original IR image in red Original Visible Image
in green Joint Segmentation
Unimodal Segmetation Unimodal Segmentation
Union of Unimodal Segmentations
39
Multi-Sensor Image Segmentation Data Set

• To enable objective comparison of different
  segmentation techniques
• Need some method of finding a ground truth of
  natural images
• The human visual system is good at segmenting
  images
• The Berkeley Segmentation Database
• 1000 natural images
• 12000 human segmentations
• Martin et al., A Database of Human Segmented
  natural Images and its Application to Evaluating
  Segmentation Algorithms and Measuring Ecological
  Statistics, ICCV, 2001

40
Multi-Sensor Image Segmentation Data Set

• 11 Sets of multi-modal images
• 14 IR and 11 grey scale images
• 33 fused images from 3 pixel-based fusion
  algorithms
• Contrast pyramids
• Discrete wavelets transform
• Dual tree complex wavelet transform
• All images have been segmented by the techniques
  described using the same good parameters across
  the whole data set

41
Image Data Set Examples
Images from the Multi-Sensor Image Segmentation
Data Set
42
Experimental Setup

• 63 subjects
• The instructions were to
• Divide each image into pieces, most important
  pieces first, where each piece represents a
  distinguished thing in the image. The number of
  things in each image is completely up to you.
  Something between 2 and 20 is usually reasonable.
  Take care and try and be as accurate as possible.
• 5 images segmented each
• Images pseudo-randomly distributed so that
• Each subject sees only one image from each set
• They see at least one IR, one visible and one
  fused image
• An image is not distributed a second time unless
  all images have been distributed once etc.

43
The Segmentation Tool
The Berkeley Segmentation Tool (SegTool)
44
The Human Segmentations

• 315 human segmentation produced
• 20 rejected as obviously wrong
• 5-6 segmentations for each image
• 1 expert segmentation for each image

The human segmentations are available to download
from www.ImageFusion.org
45
Examples of Human Segmentations
User 5
User 35
User 15
User 61
User 54
User 39
Human Segmentations of UN Camp CWT Fused Image
46
Segmentation Error Measure I

• We adopt the approach used with the Berkley
  Segmentation Dataset
• Precision, P, fraction of detections that are
  true positives rather than false positives
• Recall, R, fraction of true positives that are
  detected rather than missed
• F-measure is a weighted harmonic mean
• F PR/(aR(1- a)P)
• a 0.5 used

47
Segmentation Error Measure II

• Correspondences computed by
• Comparing the segmentation to each human
  segmentation of that image
• Correspondence computed as a minimum cost
  bipartite assignment problem
• Scores averaged to give a single P, R and F value
  for each image
• Tolerates localization errors
• Finds explicit correspondences only

48
Analysis of Human Segmentations
49
Examples of Automatic and Human Segmentations I
Images from the Multi-Sensor Image Segmentation
Data Set
50
Examples of Automatic and Human Segmentations II
Images from the Multi-Sensor Image Segmentation
Data Set
51
Joint Vs Uni-Modal Segmentation (Original Images)
52
Multi-Sensor Image Segmentation Results

• Using the human segmentations as ground truth
  for evaluation
• Found UoB_Uni to give best segmentations of
  uni-modal techniques
• Found joint segmentations to be better than the
  uni-modal segmentations of the original images
• Found the joint segmentations to be at least as
  good as the uni-modal segmentations of the fused
  images
• The relevance of these results to region-based
  fusion confirmed

Joint- versus Uni-Modal Segmentation for
Region-Based Image Fusion", J. J. Lewis, S.
G. Nikolov, A. Toet, D. R. Bull and C. N.
Canagarajah, Proceedings of Fusion 2006
53
Multi-Sensor Image Segmentation Work in Progress

• Recent results indicate that schemes for fusion
  of visible and IR imagery should prioritise
  terrain features from the visible imagery and
  man-made targets from the IR imagery in the
  fusion process, in order to produce a fused image
  that is optimally tuned to human visual cognition
  and decision making
• By comparing the human segmentations of the input
  images to the human segmentations of the fused
  images we can hopefully study how image fusion
  affects segmentation decisions

54
Summary I

• Multi-sensor image fusion affects decision making
  in various ways
• By applying tasks to the image fusion assessment
  process, it has been found that DT-CWT fusion can
  lead to better target detection human performance
  than AVE, pyramid and DWT methods
• In addition, the objective tasks utilised have
  been shown to produce very different patterns of
  results to comparative subjective tasks.

55
Summary II

• Fused videos with higher (perceived) quality do
  not necessarily lead to better tracking
  performance
• In most cases there are significant advantages of
  using a fused video sequence for target tracking
  even in HL levels and more so in LL levels
• Using the Multi-Sensor Segmentation Data Set we
  are trying to produce fused images that are
  optimally tuned to human visual cognition and
  decision making and to study how image fusion
  affects segmentation decisions

56
Acknowledgements

• NATO and the ARW organisers
• The Data and Information Fusion Defence
  Technology Centre (DIF-DTC), UK, for partially
  funding this research
• The Image Fusion Toolbox (IFT) and the Video
  Fusion Toolbox (VFT) development team at the
  University of Bristol
• Lex Toet (TNO Defence and Security, The
  Netherlands), Dave Dwyer (Octec Ltd, UK) and
  Equinox Corp (USA) for providing some of the
  images sequences used in this study (all these
  image sequences are available through
  www.ImageFusion.org)
• The Eden Project in Cornwall