Vision Guided Robotics

1
Vision Guided Robotics

• and Applications in Industry and Medicine
• Matthias Rüther

2
Contents

• Robotics in General
• Industrial Robotics
• Medical Robotics
• What can Computer Vision do for Robotics?
• Vision Sensors
• Issues / Problems
• Visual Servoing
• Application Examples
• Summary

3
Robotics

• What is a robot?
• "A reprogrammable, multifunctional manipulator
  designed to move material, parts, tools, or
  specialized devices through various programmed
  motions for the performance of a variety of
  tasks"
• Robot Institute of America, 1979
• Industrial
• Mostly automatic manipulation of rigid parts with
  well-known shape in a specially prepared
  environment.
• Medical
• Mostly semi-automatic manipulation of deformable
  objects in a naturally created, space limited
  environment.
• Field Robotics
• Autonomous control and navigation of a mobile
  vehicle in an arbitrary environment.

4
Robot vs Human

• Robot Advantages
• Strength
• Accuracy
• Speed
• Does not tire
• Does repetitive tasks
• Can Measure

• Human advantages
• Intelligence
• Flexibility
• Adaptability
• Skill
• Can Learn
• Can Estimate

5
Industrial Robot

• Requirements
• Accuracy
• Tool Quality
• Robustness
• Strength
• Speed
• Price Production Cost
• Maintenance

Production Quality
6
Medical (Surgical) Robot

• Requirements
• Safety
• Accuracy
• Reliability
• Tool Quality
• Price
• Maintenance
• Man-Machine Interface

7
What can Computer Vision do for Robotics?

• Accurate Robot-Object Positioning
• Keeping Relative Position under Movement
• Visualization / Teaching / Telerobotics
• Performing measurements
• Object Recognition
• Registration

Visual Servoing
8
Vision Sensors

• Single Perspective Camera
• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)
• Laser Scanner
• Omnidirectional Camera
• Structured Light Sensor

9
Vision Sensors

• Single Perspective Camera

10
Vision Sensors

• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)

11
Vision Sensors

• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)

12
Vision Sensors

• Laser Scanner

13
Vision Sensors

• Laser Scanner

14
Vision Sensors

• Omnidirectional Camera

15
Vision Sensors

• Omnidirectional Camera

16
Vision Sensors

• Structured Light Sensor

Figures from PRIP, TU Vienna
17
Issues/Problems of Vision Guided Robotics

• Measurement Frequency
• Measurement Uncertainty
• Occlusion, Camera Positioning
• Sensor dimensions

18
Visual Servoing

• Vision System operates in a closed control loop.
• Better Accuracy than Look and Move systems

Figures from S.Hutchinson A Tutorial on Visual
Servo Control
19
Visual Servoing

• Example Maintaining relative Object Position

Figures from P. Wunsch and G. Hirzinger.
Real-Time Visual Tracking of 3-D Objects with
Dynamic Handling of Occlusion
20
Visual Servoing

• Camera Configurations

End-Effector Mounted
Fixed
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
21
Visual Servoing

• Servoing Architectures

Figures from S.Hutchinson A Tutorial on Visual
Servo Control
22
Visual Servoing

• Position-based and Image Based control
• Position based
• Alignment in target coordinate system
• The 3D structure of the target is rconstructed
• The end-effector is tracked
• Sensitive to calibration errors
• Sensitive to reconstruction errors
• Image based
• Alignment in image coordinates
• No explicit reconstruction necessary
• Insensitive to calibration errors
• Only special problems solvable
• Depends on initial pose
• Depends on selected features

End-effector
target
Image of end effector
Image of target
23
Visual Servoing

• EOL and ECL control
• EOL endpoint open-loop only the target is
  observed by the camera
• ECL endpoint closed-loop target as well as
  end-effector are observed by the camera

EOL
ECL
24
Visual Servoing

• Position Based Algorithm
• Estimation of relative pose
• Computation of error between current pose and
  target pose
• Movement of robot
• Example point alignment

p1
p2
25
Visual Servoing

• Position based point alignment
• Goal bring e to 0 by moving p1
• e p2m p1m
• u k(p2m p1m)
• pxm is subject to the following measurement
  errors sensor position, sensor calibration,
  sensor measurement error
• pxm is independent of the following errors end
  effector position, target position

26
Visual Servoing

• Image based point alignment
• Goal bring e to 0 by moving p1
• e u1m v1m u2m v2m
• uxm, vxm is subject only to sensor measurement
  error
• uxm, vxm is independent of the following
  measurement errors sensor position, end effector
  position, sensor calibration, target position

p1
p2
u1
v1
v2
u2
d1
d2
c1
c2
27
Visual Servoing

• Example Laparoscopy

Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
28
Visual Servoing

• Example Laparoscopy

Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
29
Registration

• Registration of CAD models to scene features

Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
30
Registration

• Registration of CAD models to scene features

Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
31
Tracking

• Instrument tracking in laparoscopy

Figures from Wei A Real-time Visual Servoing
System for Laparoscopic Surgery
32
Summary

• Computer Vision provides accurate and versatile
  measurements for robotic manipulators
• With current general purpose hardware, depth and
  pose measurements can be performed in real time
• In industrial robotics, vision systems are
  deployed in a fully automated way.
• In medicine, computer vision can make more
  intelligent surgical assistants possible.