MapEnhanced UAV Image Sequence Registration and Synchronization of Multiple Image Sequences

1
Map-Enhanced UAV Image Sequence Registration and
Synchronization of Multiple Image Sequences

• Yuping Lin and Gérard Medioni

2
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

3
Introduction

• Input
• Multiple UAV video streams
• Position of moving objects in each video stream
• Goal Synchronize using a common moving object

4
Method

• Register UAV streams to a global reference image
  (a map), then
• Synchronize the streams using the unique path of
  a common moving object on the map

5
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

6
Register UAV streams to a global reference image

• Input
• Global reference image (Map)
• UAV stream
• The homography of the first frame of the UAV
  stream to the map

7
Register UAV streams to a global reference image

• Issues

• APPROACH

• UAV images and the map are different in terms of
  viewpoints, sensors and time of capture
• Direct matching is difficult

• Given the homography of the first UAV frame to
  the map,
• Two step registration
• Consecutive UAV image registration, then
• UAV to Map registration

8
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

9
Consecutive UAV Image Registration

• Method
• extract features in each frame
• Establish feature correspondences between
  consecutive images
• estimate the transformation

10
Consecutive UAV Image Registration

• Issues

• approach

• Features should be descriptive for matching and
  sufficient to give good transform estimation
• Feature matching
• Transform estimation

• SIFT feature extraction
• 128 dimension feature descriptor
• Avg. 2000 features in each image
• Nearest neighbor matching
• Avg. 1000 matches in each pair of images
• RANSAC
• Avg. 600 inliers in each pair of images

11
Consecutive UAV Image Registration

• Illustration

12
Consecutive UAV Image Registration

• Illustration

13
Consecutive UAV Image Registration

• Illustration

14
Consecutive UAV Image Registration

• Illustration

15
Consecutive UAV Image Registration

• Illustration

16
Consecutive UAV Image Registration

• Problem error is accumulated

17
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

18
UAV to Map Registration

• Method
• Perform local search for correspondences between
  the UAV image and the map

19
UAV to Map Registration

• issues

• approaches

• UAV images are very different from the map, SIFT
  features cannot always match

• Sample points in the map
• For each point, locally search for the most
  similar image patch in the UAV image
• Use Mutual Information as similarity measurement

20
UAV to Map Registration

• Illustration

21
UAV to Map Registration

• Illustration

22
UAV to Map Registration

• Illustration

23
UAV to Map Registration

• Illustration

24
UAV to Map Registration

• Illustration

25
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

26
Iterative way

• Method
• Perform consecutive UAV image registration and
  UAV to Map registration iteratively
• Consecutive UAV image registration produce good
  initials for UAV to Map registration
• Register the partial local mosaic to the map

27
Register Partial Local Mosaic

• ISSUES

• APPROACH

• Correspondences in a single frame are not
  enoughRegistration is unstable

• Multiple frames in a time window forms a partial
  local mosaic which spans a larger region and
  provides more correspondences
• More robust
• Smooth transition

28
Register Partial Local Mosaic

• ISSUES

• APPROACH

• Correspondences in a single frame are not
  enoughRegistration is unstable

• Multiple frames in a time window forms a partial
  local mosaic which spans a larger region and
  provides more correspondences
• More robust
• Smooth transition

29
Register Partial Local Mosaic

• Result

Register single frame
Register partial local mosaic
30
Iterative way

• Illustration

31
Iterative way

• Illustration

32
Iterative way

• Illustration

33
Iterative way

• Illustration

34
Register UAV streams to a global reference image

• Result

35
Outline

• Introduction
• Method
• Register UAV streams to a global reference image
• Consecutive UAV image registration
• UAV to Map registration
• Interleaving image to image and image to map
• Partial local mosaic
• Synchronization of multiple video streams
• Conclusion

36
Synchronization of Multiple Video Streams

• Input UAV image sequences of different views,
  different frame rates, but capture the same area
  and overlap in time
• An moving object on the ground plane which serves
  as a clock to synchronize the sequences

37
Synchronization of Multiple Video Streams

• The moving object should generate a single path
  on the map
• Use sequence alignment algorithm to synchronize
  the UAV streams

38
Synchronization Result
39
Conclusion

• Two steps to register an UAV image to the map
• Register each frame to its previous frame to
  derive an initial estimate
• Register UAV image to the map to derive
• Limitations
• Initial estimate should be given
• Unable to recover from a bad estimate