A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System

1
A Hierarchical Design Scheme for Application of
Augmented Reality in a Telerobotic Stereo-Vision
System
By Syed Mohammed Shamsul Islam Lecturer B and PhD
student Department of Computer Engineering KFUPM

• Co-authors
• Prof. Mayez Al-Mouhamed
• Dr. S. M. Buhari
• Dr. Talal Al Kharobi

2
Outline

• Introduction
• Design Issues
• Implementation
• Performance Evaluation
• Conclusions Recommendations

3
Telerobotics

• A scheme that allows humans to extend their
  manipulative skills over a network.

Visual feedback
Visual feedback
Force feedback
Force feedback
4
Telerobotic Applications

• Hazardous situations
• e.g. bomb disposal
• Scaled-down and scaled-up situations
• e.g. micro-surgery
• Situations affected by human presence
• e.g. closed-chest heart bypass
• Teaching, training, maintenance, entertainment.

5
Problem of Conventional Tele-robotics

• Time delay
• Processing, copying video data, network delays
• Move-n-Wait strategy
• Operator has to follow trial and error method to
  perform a task, which might be dangerous for
  example in telesurgery.

6
Improving Performance by Using AR

• By overlaying virtual image with the real image
  operator e.g. surgeon can make a plan before
  going for actual teleoperation ? safety.
• Rehearsal and correction can be made in the
  simulation plan.
• Saving bandwidth by sending (less frequently)
  only the planned data.

7
Challenges of AR

• Graphical robot arm must mach with real robot arm
  in the video.

8
Challenges of AR

• Camera Calibration
• Finding accurate camera calibration is
  challenging.
• Viewpoint Registration

9

• Design Issues

10
Telerobotic Stereo-Vision System to be Augmented
11
Design Methodology
12
Mathematical Model of the Robot Manipulator

• PUMA 560 Slave Arm
• 6 DOF
• 6 links are interconnected
• serially to each other except
• first and last.
• All the joints are rotational.

A rotational joint
13
Mathematical Model of the Robot Manipulator

• Each link of the arm is attached with a frame of
  reference Ri (xi,yi,zi) to specify its position
  and orientation.
• Link Li1 rotates w.r.t Li when frame Ri1
  rotates w.r.t either axes of xi, yi, or zi
• Link vector OiOi1,i can be expressed as

• Now position and orientation of Oi1 w.r.t Oi-1
  can be expressed as

14
Mathematical Model of the Robot Arm

• Position and orientation of the end effector is
  expressed relative to the frame of the base.
• So the geometric model of robot arm can be
  expressed as

This model will give us a skeleton of the
graphical arm.
15
Building Body Shapes Around the Skeleton of
Graphical Arm

• Alternatives
• Taking as is link 3 and 4 are trapezoidal,
  others are cylindrical.
• Considering all as cylindrical

16
Building Body Shapes Around the Skeleton of
Graphical Arm

• Cylinder is drawn with finite element
  representation.
• Increasing number of segments will improve
  quality of view but increase computational
  complexity.
• Alternatives of primitive for drawing a cylinder
• Triangles
• Triangle stripe, Triangle fan, Triangle list
• Lines
• Line stripe, Line list

17
Data Structure Design

• Types of data and nature of manipulation
• A link is represented by its start and end point
  co-ordinates, radius, axis of rotation, joint
  angle, orientation matrix etc.
• Each link is also associated with the vertices of
  its body shape.
• Links are serially connected and position and
  orientation of a link depends on position and
  orientation of its previous link.
• There is frequent movement of links in
  teleoperation and each movement involves matrix
  calculation for new position and orientation of
  the links to be changed.
• Objectives
• To reduce processing time by reducing
  computation.
• To provide flexibility and generality

18
Data Structure Design

• Our strategy
• Each link is defined as an object ? geometric
  data of each link is now apart from the other
  link,.
• Configuration data (number of links, number of
  segments in cylinder etc) kept in separate data
  file.
• World and view transformation matrices are
  combined to reduce matrix multiplication.
• Vertex data of body shape of each link is stored
  in separate vertex buffer.

19
Displaying the Graphical Arm

• Given the robot structure in its model space
• Rendering on the display screen
• Scene layout setup
• Transformation matirces, Camera position, target
  position, viewing angle, lighting.
• Viewing Models
• Solid Model or Wire-frame Model
• Psychological studies and one performance
  studies in telerobotic system of shows no
  significant diff.
• Wire-frame models have some adv.-reduce
  occlusion, lesser rendering time.
• Rendering
• Scanline rendering, Z-buffering algorithm for
  HSR.

20
Sample Display of Graphical Arm
Wire-frame view
Solid view
21
Movement of Graphical Arm in Joint Space
22
Movement of Graphical Arm in Cartesian Space
Start
Receive the incremental position (?X) and
orientation (?M) matrices
Calculate new position, Xnew(t) and orientation,
Mnew(t) matrices based on current frame of
reference
Calculate corresponding angular values Using
Inverse Geometric Transformation (?newG-1(Xnew,
Mnew)
Angle within bound?
B
23
Movement of Graphical Arm in Cartesian Space
24
Acquisition of Real Video Image

• The MDTF Approach is used for stereovision.
• It is multi-threaded, distributed and provide
  better performance in video transfer.
• Real video image of the slave robot at the server
  side is captured simultaneously by two video
  cameras.
• Then a reliable client-server connection is
  established and upon a request from the client a
  stereo frame comprising of two pictures is sent
  over LAN through window sockets.
• On the client side after detecting and making
  connection with the server pictures are received
  and displayed on the screen and on the HMD when
  connected.
• A double buffer, concurrent transfer approach is
  used to maximize overlapped transfer activities
  between cameras, processor and the network.

25
3D Visualization Tools and Techniques

• Technique
• Alternatives Sync-Doubling, Page-flipping
• Chosen Page-flipping
• Provides higher resolution and avoid flashing
  problem of 3D imaging.
• Display device
• Alternatives HMD, monitor, eye-shuttering glass
• Chosen HMD, monitor
• No flickering, easy to work

26
3D Visualization Example
A snap shot of a stereo-image.
27
Camera Calibration

• Heikkilas calibration method which provides very
  accurate results. It shows accuracy up to 1/50 of
  the pixel size.
• Pinhole camera model with perspective projection
  and least-square error optimization.
• The intrinsic camera parameters
• Focal length, fc
• Principal point, cc
• Skew co-efficient, alpha_c
• Lens distortion co-efficient, kc

28
Camera Calibration
The aspect ratio fc(2) / fc(1) The field of view
angle can be calculated from,
29
Camera Calibration

• Extrinsic Parameters

If XXX,Y,Z and XXc are co-ordinate of P in
grid and camera ref, then
Where, translation vector Tc_1 is the co-ordinate
vector of O in camera ref frame and the rotation
matrix Rc_1 is the surface normal vector of the
grid plane in the camera ref frame.
30
3D Visualization Algorithm

• Step 1 Acquire video image and copy this memory
  stream to a surface( say FrontSurf).
• Step 2 Copy the FrontSurf surface to another
  temporary surface (say, AugSurf).
• Step 3 Draw graphical objects/change in
  graphical objects on AugSurf.
• Step 4 Copy AugSurf to the surface that will be
  used to display (say, backSurf).
• Step 5 Display backSurf with left camera image
  to left view port/monitor.
• Step 6 Display backSurf with right image to
  right view port/monitor.
• Step 7 Observe 3D with HMD.

31
Graphical Tele-manipulation

• How Graphic Manipulation Corresponds to the Real
• If the base link of the real robot remains fixed
  relative to the video cameras, the base link of
  the graphical arm will also remain fixed relative
  to the graphical cameras.
• The end-effector of the graphical arm can be
  manipulated in the graphical coordinate space,
  relative to objects in the task space (keeping
  base link in same location of real robot base).

32
GUI Design

• Client Side Input/Output Specification
• Connecting to the server PC, master arm and HMD
• Receiving and displaying the stereo video.
• Taking user's input for movement of the real and
  graphical robot for simulation
• Means of User Interaction
• Master arm
• Joystick
• Keypad
• Mouse

33

• Implementation

34
Tools Used

• Client-server communication platform
• MS .NET with .NET Remoting
• Programming Language
• Visual C.NET
• Graphics Tool
• Microsoft DirectX
• 3D Graphics API
• Alternatives
• Windows GDI
• Java3D by Sun MicroSystems
• Open GL (Open Graphics Language) by SGI
  Silicon Graphics
• Direct3D

35
Motivation for Using Direct3D

• It increases performance by using hardware
  acceleration.
• It allows applications to run full-screen instead
  of embedded in a window. As we are using HMD this
  feature is very helpful in our case to have the
  3D effect.
• Direct3D perfectly match with Microsoft's various
  Windows operating systems and with Microsoft .NET
  framework which are used in our telerobotic
  client-server framework

36
Direct3D Architecture
37
Graphics Implementation

• Virtual Object Modeling Module
• Defines the structure of the virtual objects
  (robot, cubes etc.)
• Handles functions like movement of virtual
  objects.
• Display Module
• Synchronization of real and virtual data
• Projection on video surface
• Augmentation of real video
• Page Flipping for HMD stereo visualization

38
Interfacing to Telerobotic Stereovision System

• Client Modules

39
GUI Implementation

• Main Form

40
GUI Implementation

• Stereo Form

41

• Performance Evaluation

42
Evaluation of Graphical System

• Refresh Rate

Environment Graphics Complexities Avg. Refresh Rate (frame/sec)
Graphics without overlaying real video image Without any drawing 273.36
Graphics without overlaying real video image With an object and the graphical arm in the scene 243.74
Graphics without overlaying real video image Only graphical arm with 8 segments in each cylinder 253.59
Graphics without overlaying real video image Only graphical arm with 50 segments in each cylinder (solid view) 239.78
Graphics without overlaying real video image Without any drawing 11.498
Graphics without overlaying real video image With an object and the graphical arm in the scene 11.384
Graphics without overlaying real video image Only graphical arm with 8 segments in each cylinder 11.347
Graphics without overlaying real video image Only graphical arm with 50 segments in each cylinder (solid view) 11.325
Average is taken over 1000 samples, running on
Pentium-4, 2GHz machine with 1GB RAM
43
Time Required for Rendering Graphical Arm
Environment Graphics Complexities Rendering Time (ms) Rendering Time (ms)
Environment Graphics Complexities Wire-frame Solid
Graphics without overlaying real video image Only graphical arm with 8 segments in each cylinder 64.088 64.283
Graphics without overlaying real video image Only graphical arm with 50 segments in each cylinder 65.034 65.151
Graphics overlaying real video image Only graphical arm with 8 segments in each cylinder 105.02 108.037
Graphics overlaying real video image Only graphical arm with 50 segments in each cylinder 105.37 109.362
Average is taken over 1000 samples, running on
Pentium-4, 2GHz machine with 1GB RAM
44
Accuracy

• Re-projection error with the calibration method
• Pixel Error (0.11689,0.11500)

Pixel Axis in the y direction
Pixel Axis in the x direction
45
Comparison to Other Apporaches

• Iqbal, A. 1 augmented with only a small red
  ball at the position of gripper in comparison to
  our whole graphical arm.
• Iqbal, A. 1 used Faugeras 4 calibration with
  Kuno5s affine frame of reference which led him
  to noticeable mismatch with real error in the
  matching shown in his figure. Whereas our
  computer vision-based calibration reduces error
  upto 1/50 of pixel size.
• J. Vallino reports in his PhD thesis refresh rate
  of 10fps to be required for AR, when we gets
  above 11-17fps after overlaying graphical arm
  with live stereo video.
• Graphics rendering of our system is faster than
  Iqbal, A. 1,23 for our use of Direct3D.
• Our system is comparatively cheaper due to the
  use of commodity hardware (PC) and software.

46

• Conclusion

47
Summary of the Work and Contributions

• Using hardware accelerated graphics rendition
  that provides us with excellent refresh rate of
  the output screen.
• Improvement in accuracy of execution by using
  better calibration method and graphic aids
  (accuracy up to 1/50 of pixel size).
• User-friendly graphical user interface for simple
  manipulation in the telerobotic AR system.

48
Summary of the Work and Contributions

• Flexible and generalized data structure suitable
  for telerobotic visualization.
• Identifying design strategy for intelligent
  switching in VR and AR mode for ensuring QoS.
• Use of cheap and commercially available hardware
  and software
• Can be used as a cheap and flexible visual tool
  for showing robot manipulation in the classrooms.

49
Future Research Directions

• Providing an intelligent system to switch between
  VR and AR modes of operation based on network
  delays to ensure QoS.
• Using multi-processor system for processing video
  and graphics data more efficiently.
• Using commercial software available to extract
  exact 3D model of the workspace objects which
  will facilitate more accurate task manipulation.

50
References

• Iqbal, A. Multistream realtime control of a
  distributed telerobotic system. M.Sc. Thesis,
  King Fahd University of Petroeum and
  Minerals,June 2003.
• A Rastogi, P. Milgram, and D. Drascic.
  Telerobotic control with stereoscopic augmented
  reality. SPIE, Vol.2653 Stereoscopic Displays
  and Virtual Reality Systems III135146, Feb.
  1996.
• R. Marin, P.J. Sanz, and J.S. Sanchez. A very
  high level interface to teleoperate a robot via
  web including augmented reality. Proc.
  IEEEInternational Conference on Robotics and
  Automation, 2002 ICRA '02, Vol.32725 - 2730, May
  2002.
• O.D. Faugeras and G. Toscani. The calibration
  problem for stereo. Proceedings of Conference on
  Computer Vision and Pattern Recognition, Miami
  Beach, FL, Vol. 5, No. 3, June, 15-20 1986.

51
References

• 5Y. Kuno, K. Hayashi, K.H. Jo, and Y. Shirai.
  Human-robot interface using uncalibrated stereo
  vision. International Conference on Intelligent
  Robots and Systems 95, 1525530, 1995.
• 6J. Abdullah and K. Martinez. Camera
  self-calibration for the ar-toolkit. The First
  IEEE International Workshop on Augmented Reality
  Toolkit, page 5, Sept. 2002.
• 7J.-Y. Herve, C. Duchesne, and V. Pradines.
  Dynamic registration for augmented reality in
  telerobotics applications. IEEE International
  Conference on Systems, Man, and Cyberneticsn,
  Vol.21348-1353, October 2000.

52
Thank You
Questions are welcome