MULTISTREAM REAL-TIME CONTROL OF A DISTRIBUTED TELEROBOTIC SYSTEM

1
MULTISTREAM REAL-TIME CONTROL OF A DISTRIBUTED
TELEROBOTIC SYSTEM
Asif Aqbal
2
Contents

• Introduction
• Background
• Status Of The Problem
• Literature Review
• Thesis Objectives
• Video Client-Server Framework
• Distributed Telerobotic Framework
• Augmented Reality
• Conclusions
• Thesis Contributions
• Future Research Directions

3
Introduction

• Telerobotics humans to extend their manipulative
  skills over a distance, extend eye-hand motion
  coordination.
• Telerobotic applications
• Scaled-down nano-scale, micro-surgery,
  clean-room
• Hazardous nuclear decommissioning inspection,
  fire fighting, disposal of dangerous objects,
  minefield clearance, operation in harsh
  environments, unmanned, underwater, ice, desert,
  space,
• Safety rescue,
• Security surveillance, reconnaissance,
• Unmanned oil platform inspection, repair,
• Teaching, training, and entertainment.

4
Introduction (cont.)

• Minefield clearance, unmanned underwater
  inspection, and search rescue.
• Those where humans adversely affect the
  environment such as medical applications and
  clean-room operations.
• Those which are impossible for humans to be
  situated in such as deep space and nanorobotics.

5
Introduction (cont.)

• Extending eye-hand motion coordination using
  telerobotics
• In natural eye-hand motion coordination, operator
  sees his hand and react accordingly.
• In telerobotics
• Operator holds a master arm to dictate his hand
  motion,
• Motion is transmitted to a remote slave arm and
  reproduced (replica),
• Operator wears a head-mounted display (HMD) to
  see in 3D the effects of his motion on the remote
  tool,
• Operator does not see his hand (HMD) nor the
  master arm, his hand is logically mapped to the
  remote tool,
• Operator logically acts on the remote tool seen
  through the HMD.
• A step further, stereo vision provides 3D views
  of slave scene to the operator. It also provides
  a metric to calculate 3D positions and
  orientations of the physical object present in
  the remote world.

6
Background

• Computer Systems Engineering integrating
  computer software and architecture, control, and
  mechanics.
• Software real-time client-server, RT-OS,
  distributed-components, multi-threading, thread
  engineering,
• Architecture/compiler parallel streaming buses,
  parallel processing, interfacing, stereo-vision,
  embedding,
• Control stability vs delay, manual, supervisory,
  automatic, position/force, shared control,
  impedance,
• Mechanical light, ergonomic, wire-based
  transmission, backdrivable, dexterous, force
  fidelity, maintainable.

7

• Visual feedback from the remote side plays an
  important role in increasing the efficiency of
  the operator tele-manipulating the robot.
• Supervisory Control as defined by Sheridan1986
  Supervisory control means that one or more human
  operators are intermittently programming and
  continually receiving information from a computer
  that itself closes an autonomous control loop
  through artificial effectors to the controlled
  process or task environment.

8
Background (cont.)

• Specifically a Telerobotic system consists of
• A master arm connected to a client program that
  monitors the operator motion and interacts with a
  slave station to ensure that the slave arm motion
  is a replica of operator motion.
• A slave arm connected to a server program that is
  capable of converting a relative operator motion
  into an incremental slave motion.

9
Background (cont.)

• A two-way logical communication link to transfer
  commands from client to the server through a
  Computer Network and to convey LAN
• different kinds of feedback, e.g., video, force
  etc., back to the client site.

10
Background (cont.)

• A Telepresence system is one which displays high
  quality information from the remote world, visual
  or otherwise, in such a natural way that the
  operator feels physically present at the remote
  site.
• Virtual Reality (VR) is the interactive
  simulation of a real or imagined environment that
  can be experienced visually or otherwise in the
  three dimensions of width, height, and depth.

11
Background (cont.)

• Augmented Reality (AR) superimposes graphics and
  virtual objects to a stereo view of slave site in
  an attempt to provide the operator, an
  anticipated view of his actions ahead of time
  which is useful to overcome overall system
  delays.
• Or it is a variation of VR in a sense that AR
  supplements reality while running the interactive
  VR simulation.

12
Background (cont.)

• Kinesthesis are the sensations derived from
  muscles, tendons and joints and stimulated by
  movement and tension.
• A haptic device is one that displays the
  kinesthesis to a human hand or arm.

13
Status Of The Problem

• Responsiveness, real-time, and quality-of-service
  are needed to provide high-quality interactions
  between operator and slave arm.
• Issues in remote sensing of the slave environment
  e.g., stereo video, force sensing, sound etc.
• Faster acquisition of video data, packing,
  zero-copy, transmission, etc.
• Camera identification problems.

14
Status Of The Problem

• Fair allocation of processor usage to multiple
  threads for control, sensors, GUI, and windows
  sockets.
• Force sensing using high precision sensors,
  passive and active compliance, force control,
  impedance control, and mixed position-force
  control.

15
Status Of The Problem

• Time delay problems.
• Processing, copying video data, network delays.
• Internet and LANs dont provide the Q.o.S.
  guarantee.
• Manual control of the remote manipulator is
  impeded if there is a time delay between command
  and feedback. Operator switches to Move-n-Wait
  strategy.
• Can drive continuous closed loop systems to
  unstability.

16
Literature Review

• Supervisory Control
• Shridan1989
• Mathew1995
• Christian1996
• Chen1997
• Ren2000

• Stereo Vision and Augmented Reality
• Dave1994
• Kuna1995
• Milgram1995
• Spyros1999

• Network Telerobotics
• Paolucci1996
• Wayne1996
• Kazuhiro1996
• Yeuk1999
• Yeuk2000

17
Thesis Objectives

• Advanced Telerobotic Software
• To develop a fully Distributed Object Oriented
  Framework using the most advanced software
  authoring tools (.NET and SOAP technologies) for
  real-time control of Robot-Server-Client
  interactions including upgrading of previously
  implemented kinematic and inverse-kinematic
  models.
• Investigation of impact of .NET component based
  approach on network time delays while deploying a
  multi-stream, multi-threaded distributed
  application having command, force and video
  streams at the same time.

18
Thesis Objectives

• Kinesthetic Mapping of Master-Slave Interaction
• Implementing geometric working frames, like tool
  and robot frames to improve the mapping between
  master and slave arms.
• Implementing a scalable mapping between master
  and slave spaces that can be controlled by the
  operator in real-time.
• Providing the operator with force-feedback
  display by using real-time stream of force
  information.

19
Thesis Objectives

• Computer Vision
• Development of an optimized client-server
  framework for grabbing, transmission and display
  of 3D stereo data over a LAN. 3D effects will be
  produced using different methods like
  sync-doubling, page-flipping etc.
• Investigation of Augmented Reality as part of a
  strategy to reduce network delays in a
  telerobotic environment by using simple pointer
  in the stereo image at the client side.

20
Video Client-Server Framework

• The provision of stereo video on the client side
  imposes severe requirements in terms of bandwidth
  to transfer real-time stream of video data in a
  telerobotic environment.
• It requires the use of advanced technologies like
  DirectX and Windows Sockets to accomplish the
  capturing and relaying of video data over a LAN.
• Commercially available software like Microsoft
  NetMeeting are optimized for a low band-width
  network like internet so they show too poor
  display resolution to be used for stereo vision
  in a telerobotic setup.

21
Video Client-Server Framework

• Development of a highly optimized client-server
  framework for grabbing and relaying of a stereo
  video stream becomes inevitable for an efficient
  telerobotic framework.
• This framework must accomplish following tasks on
  the server side
• Capture or grab stereo images from two cameras at
  the slave side simultaneously.
• Establish a reliable client-server connection
  over a LAN, the slave side being the vision
  server.
• Upon requests from the client send this stereo
  frame comprising of two pictures to the client
  through windows sockets.

22
Video Client-Server Framework

• On the client side it should provide the
  following functionality
• Establish a highly optimized fast graphic display
  system to show the pictures received from the
  server.
• Detect and establish the connection with server.
• Display the pictures arrived from the server and
  continue in a loop each time asking a new stereo
  frame from the server.
• Allow the viewer to adjust the alignment of the
  pictures on the output device, whatever it is, to
  compensate for the misalignment and
  non-linearities present in the stereo camera
  setup at server side.

23
Video Client-Server Framework

• A client-server framework fulfilling the above
  defined requirements is developed using the
  advanced software development tools like
  Microsoft Visual C and Microsoft DirectX.
• Microsoft DirectX provides COM based interfaces
  for various graphics related functionalities.
  DirectShow is one of these services. DirectShow,
  further, provides efficient interfaces for the
  capturing and playback of video data.

24
Video Client-Server Framework

• We can use network services and send/receive data
  over a network using windows sockets. The stereo
  video setup uses synchronous windows sockets as
  an interface between vision server and client.
• Two different schemes were implemented to
  transfer the video data. The schemes differ in
  the usage of multiple threads on the server side
  as well as some optimization steps to reduce the
  network traffic for the transfer of the video
  data.
• A general overview of the image grabbing and
  displaying system is given before the detailed
  description of the above scheme.

25
Video Client-Server Framework

• We use a component of DirectShow named
  SampleGrabber to capture video frames coming
  through a stream from a stereo camera setup. A
  block diagram of the scheme used at the server
  side to grab stereo frames is shown below

26
Video Client-Server Framework

• In order to show the received pictures from the
  server, we need to use GDI (Graphics Device
  Interface). A block diagram of the client side
  scheme to display the video is shown below

27
Video Client-Server Framework(Single Buffer,
Serialized Transfer)
28
Video Client-Server FrameworkDouble Buffer,
De-Serialized Transfer

• In this scheme, we try to optimize the transfer
  of video data over the LAN by using thread
  manipulation on the server.
• Thread overlapping among capture and sending
  thread is achieved using double buffers on the
  server side.
• It is ensured that the thread responsible for
  sending the video data over the LAN will not wait
  after receiving a picture request from the client.

29
Video Client-Server FrameworkDouble Buffer,
De-Serialized Transfer
30
Video Client-Server FrameworkDouble Buffer,
De-Serialized Transfer

• This approach enables us to send higher number of
  stereo frames over the same LAN and hardware.
• The only overhead is the allocation of extra
  buffer in the server DRAM which not a real
  problem with available systems containing large
  memory.

31
Video Client-Server Framework3D Visualization

• There can be different methods to produce 3D
  effects on the client side once we have stereo
  images of the remote scene.
• Similarly different hardware device such as
  eye-shuttering glasses, HMD (Head Mounted
  Display) are used to show the images to the user.
• We have used following two methods for stereo
  image production on client side
• Sync-Doubling
• Page Flipping

32
Video Client-Server FrameworkSync-Doubling

• Left and right eye images are arranged in an up
  and down way on the computer screen.
• A sync-doubler sits between the display output
  from the PC and the monitor to insert an
  additional frame v-sync between the left and
  right frames (i.e. the top and bottom frames).
• This will allow the left and right eye images to
  appear in an interlaced pattern on screen.
• Using the frame v-sync as the shutter alternating
  sync allows us to synchronically transmit the
  right and left frames to respective left and
  right eyes, thus creating a three-dimensional
  image.

33
Video Client-Server FrameworkSync-Doubling
34
Video Client-Server FrameworkPage Flipping

• Page-flipping means alternately showing the left
  and right eye images on the screen.
• Combining the 3D shuttering glasses with this
  type of 3D presentation requires the application
  of frame v-sync as the shutter alternating sync
  to create a 3D image.
• HMD can also be used in a way that two different
  images are sent on two different LCD screens of
  the HMD. The user sees the different image for
  both eyes thus feeling the depth of the scene.
  DirectX can be used to flip both the images
  simultaneously.

35
Video Client-Server FrameworkPerformance
Evaluation

• Different experiments were conducted to test the
  visual quality of the client-server setup as well
  as find the time delays and other measures of the
  video data.
• The specifications of the stereo frame are as
  under
• Height of each picture 288 pixels
• Width of each picture 360 pixels
• Size 304 KB (311040 Bytes) per picture
• 608 KB (622080 Bytes) per stereo
  frame
• Each stereo frame is of size 0.6 MB and requires
  a bandwidth of about 5Mbps/Frame on the LAN. This
  simple calculation shows the limitation of the
  100 Mbps LAN to transfer only 20 fps at the
  highest possible transfer rate.

36
Video Client-Server FrameworkPerformance
Evaluation

• Copying from SampleGrabber to DRAM
• Case 1 Copy times on server Single Force
  Thread
• 300 stereo frames
• Mean value 24.025 ms
• 95 CI between 23.29 ms and 24.75 ms.

37
Video Client-Server FrameworkPerformance
Evaluation

• Copying from SampleGrabber to DRAM
• Case 2 Copy times on server - Two Threads
• 300 stereo frames
• Mean value 60.48 ms
• 95 CI between 8 ms and 150 ms.

38
Video Client-Server FrameworkPerformance
Evaluation

• Copying from SampleGrabber to DRAM
• Case 3 Copy times on server with Force transfer
  over LAN
• 300 stereo frames
• Mean value 33.46 ms
• 9.43 ms additional for adding network transport
  thread.

39
Video Client-Server FrameworkPerformance
Evaluation

• Transferring over the LAN
• Case 1 Single Buffer, Serialized Transfer
• 300 stereo frames
• Mean value 86.1 ms
• 11.61 stereo frames/second.

40
Video Client-Server FrameworkPerformance
Evaluation

• Transferring over the LAN
• Case 2 Double Buffer, De-Serialized Transfer
• 60,000 stereo frames
• Mean value 58.94 ms
• 17 stereo frames/second.
• 90 CI between 56.0 and 64.8 ms.

41
Video Client-Server FrameworkResults Summary
Scheme Cameras to Server DRAM (ms) Server to Client (ms) Frames Per Second
Single Buffer, Serialized 24.025 86.1 11.61
Double Buffer, De-serialized 24.025 58.94 17

• Housheng et. al.2001 reported a transfer rate
  of 9-12 fps for a compressed single image of size
  200X150 pixels over a LAN. While our scheme
  transfers 17-18 uncompressed stereo fps of size
  360X288 pixels each.
• Network bandwidth is near saturated with 18 fps.

42
A Multi-threaded Distributed Telerobotic Framework

• Distributed application programming is one of the
  different schemes to establish a reliable
  connection between master and slave arms.
• Different items are realized as software
  components and then these components communicate
  with each other using distributed components
  paradigm.
• Object Oriented Approach
• Software reusability
• Easy extensibility
• One time debugging
• Multi-user environment
• Data encapsulation

43
A Multi-threaded Distributed Telerobotic Framework

• By using the distributed programming, network
  protocol issues can be avoided. The distributed
  framework itself takes care of all the network
  resources and binary data transfer over the
  network.
• Previously DCOM (Distributed Component Object
  Model) based components have been used in
  telerobotics by Yeuk et. al.
• .NET components are more advanced than COM based
  components and offer complete support of .NET
  framework including .NET Remoting and SOAP
  technologies.
• Several components are developed on server as
  well as client side and will be explained briefly.

44
A Multi-threaded Distributed Telerobotic
Framework Server Side Components

• PUMA Component
• Force (Sensor) Component
• DecisionServer Component

45
A Multi-threaded Distributed Telerobotic
Framework Server Side Components

• PUMA Component
• PUMA component is the heart of the distributed
  framework as this deals with the commands being
  sent to robot and the response of the robot.
• This component acts as a software proxy of the
  robot.
• Commands are issued to the component as they are
  being issued to the robot. Whenever robot, under
  the action of these commands, changes its states,
  the component updates itself automatically to
  reflet these changes.

46
A Multi-threaded Distributed Telerobotic
Framework PUMA Component
47
A Multi-threaded Distributed Telerobotic
Framework PUMA Component

• PUMA offeres public methods like
• InitializeRobot, to initialize the robot to a
  specific position or orientation.
• MoveIncremental, to move the robot incrementally
  in either joint or cartesian space. Overloaded.
• MoveAbs, moves the robot to the absolute
  position supplied as an argument in joint or
  cartesian space. Overloaded.

48
A Multi-threaded Distributed Telerobotic
Framework PUMA Component

• The component also provides access to many useful
  values through public properties like
  CurrentOrienationMatrix, CurrentPositionVecotr,
  CurrentJointAngles, RS-232Settings,
  CurrentWorkingFrameMode, etc.
• In addition, the component invokes different
  events conveying the changes in robot states like
  RobotMoved, DataReceived, StatusChanged,
  ErrorOccured, etc.
• Robot status is described by 6 discrete states,
  the change in which is notified by StatusChanged
  event.

49
A Multi-threaded Distributed Telerobotic
Framework PUMA move in joint space
Any No will cause a FALSE return.
50
A Multi-threaded Distributed Telerobotic
Framework PUMA move in Cartesian space
Any No will cause a FALSE return.
51
A Multi-threaded Distributed Telerobotic
Framework Server Side Components

• Force Sensor Component
• This component takes care of the force and/or
  position sensing operations carried out on the
  server as well as client sides.
• Force sensing operation is done in a dedicated
  read thread, the priority of which can be
  adjusted during runtime to allow for the better
  management of CPU usage.
• Due to multi-threading, the Force Sensor
  component supports both the reading and writing
  operations simultaneously.

52
A Multi-threaded Distributed Telerobotic
Framework Force Sensor Component
53
A Multi-threaded Distributed Telerobotic
Framework Force Sensor Component

• Force Component offers public methods like
• StartReading, creates a separate reading thread
  with default thread priority and sampling
  interval.
• StopReading, stops and aborts the force reading
  thread.
• The component exposes public properties like
• SensorThreadPriority, to set sensor thread
  priority.
• TimerValue, to set the sampling interval.
• ThresholdValue, sets the minimum change in force
  value required to invoke the ForceDataRead event.

54
A Multi-threaded Distributed Telerobotic
Framework Server Side Components

• DecisionServer Component
• An inherited component possessing the
  functionalities of both PUMA and Force Sensor
  components and adding more to it.
• To remotely access the public members of PUMA and
  Force Sensor components using IDecisionServer
  interface in order to realize a truly distributed
  framework.
• To implement a supervisory control and learning
  mode in a higher abstraction layer during
  possible future work.
• To Implement different modifiers to the commands
  coming from the client, e.g. workspace
  scalability function.

55
A Multi-threaded Distributed Telerobotic
Framework DecisionServer Component
56
A Multi-threaded Distributed Telerobotic
Framework Server Side Components

• Server Side Interfaces and .NET Remoting
• An interface is a set of public methods and
  definitions. It serves as a contract for any
  component that implements this interface.
• All the server side components must implement the
  interface IDecisionServer which in turn is
  inherited from IForce and IProxyRobot.
• This scheme is required because .NET Remoting (a
  component of .NET framework) requires the
  interfaces on client side to provide references
  to the server side objects.

57
A Multi-threaded Distributed Telerobotic
Framework Client Side Components

• IDecisionServer Interface
• MasterArm Component

58
A Multi-threaded Distributed Telerobotic
Framework Client Side Components

• IDecisionServer Interface
• In the start carries only an un-referenced
  interface to DecisionServer component.
• After establishing a network connection with the
  server, the client gets reference to server side
  instance of DecisionServer component through
  IDecisionServer interface.
• Client side can access the methods and properties
  of DecisionServer similar to invoking them on a
  local component.

59
A Multi-threaded Distributed Telerobotic
Framework Client Side Components

• MasterArm Component
• Implements all the functionality required to
  interact with a force feedback master arm in
  real-time.
• After initialization it has an active instance
  of Force component with two active threads, one
  each for reading and writing position/force
  information.
• It also implements the local force feedback to
  facilitate the operator by reducing the amount of
  force required to manipulate the master arm
  similar to a power steering mode.

60
A Multi-threaded Distributed Telerobotic
Framework MasterArm Component

• Local force feedback uses a second order model
  for minimizing the force applied by the operator.
• In order to estimate the force, the component
  maintains a record of all the force data read for
  a certain number of samples (history) along with
  the record of the system time.
• Then it evaluates the velocity and acceleration
  of the master arm at each sampling instant and
  stores them in a circular buffer.
• This information is used to calculate the force
  proportional to what the operator is applying
  which is then fed back to the master arm.

61
A Multi-threaded Distributed Telerobotic
Framework MasterArm Component
62
A Multi-threaded Distributed Telerobotic
Framework MasterArm Component

• Some of the public methods exposed by MasterArm
  component are
• StartReading, starts reading the position data
  from the master arm. Inherited from Force
  component.
• StopReading, stops reading position data.
• WriteForceData, writes supplied force data to
  output channels for force output device.
• The component exposes public properties like
• Incremental Orientation Matrix, provides the
  incremental orientation matrix when new position
  data is read.
• Incremental Position Vector, provides the
  incremental position vector.

63
A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme

• To receive the DecisionServer events on client
  side for further processing, DecisionServer needs
  the client assembly to be visible.
• Violates the object oriented philosophy.
• Restricts the deployment to multi-user
  environments.
• Shim classes are used as intermediate agents to
  forward server side events to the client side.
  They are compiled in a common library component
  visible to both server and client.

64
A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme
Server Side
65
A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme
Client Side
66
A Multi-threaded Distributed Telerobotic System

• With all the client and server components, we now
  proceed to a full-blown distributed telerobotic
  system
• This is multithreaded because more than one
  thread are running on the client and server for
  different tasks like image grabbing,
  reading/writing force sensors, sending control
  signals to robot, etc.
• Transferring real-time streams of force, command
  and stereo video over LAN makes it a
  multi-streaming environment.
• The logic of the framework is distributed among
  different hardware and software components.

67
A Multi-threaded Distributed Telerobotic System
Server
68
A Multi-threaded Distributed Telerobotic System
Client
69
Client GUI
70
A Multi-threaded Distributed Telerobotic System
Performance Evaluation

• LAN Bandwidth 100 Mbps
• Client and Server located in the same lab
• Client and Server 2.0 Ghz P-IV machines.
• 1 GB DRAM
• Accelerated Graphics Board on Client
• Each force packet 6 double values
• 48 bytes

71
A Multi-threaded Distributed Telerobotic System
Performance Evaluation

• Force transfer only
• 3000 force packets.
• Mean inter-arrival time 0.679 ms
• 90 CI between 0.59 to 0.92 ms.
• Worst case inter-arrival 825 ms.

72
A Multi-threaded Distributed Telerobotic System
Performance Evaluation

• Force and video streams
• 3000 force packets.
• Mean inter-arrival time 1.08 ms
• An addition of 0.4 ms.
• 90 CI between 0.5 and 3.9 ms.
• Worst case inter-arrival 789.74 ms.

• During the transfer of video data
• 3710 force packets.
• Mean inter-arrival time 3.9 ms
• 90 CI between 0.5 and 13 ms.

73
A Multi-threaded Distributed Telerobotic System
Performance Evaluation
74
A Multi-threaded Distributed Telerobotic System
Performance Evaluation

• A magnified plot of inter-arrival times in the
  presence of force, video and command streams.

75
A Multi-threaded Distributed Telerobotic System
A comparison

• Teresa1999 developed JAVA and VRML based
  telerobotic system and reported a image
  acquisition time of 1s for one single frame of 16
  bit depth. Our DirectShow based system reports a
  24 ms stereo image acquisition time in a
  telerobotic system.
• Al-Harthy2001 implemented client-server
  framework takes around 50ms to transfer a command
  signal (48 bytes) from client to robot. In our
  case a similar packet (48 bytes) takes from 0.7
  to 1.1 ms due to the efficient utilization of raw
  network resources by .NET Remoting.

76
A Multi-threaded Distributed Telerobotic System
Videos

• Lifting a 100 gram weight

77
A Multi-threaded Distributed Telerobotic System
Videos

• Pressing and maintaining the contact on a spring

78
Augmented Reality

• The basic idea of an AR (augmented reality)
  reality system is to mix the real and virtual
  information in order to provide an augmented view
  of the remote scene that provides more
  information than a simple video could offer.
• AR can be used as an effective way to overcome
  the effects of time delays in a telerobotic
  environment.
• The information added locally must fit seamlessly
  into the remote real data so as to avoid any
  perplexities for the teleoperator.

79
Augmented Reality Work Strategy

• To introduce non-existent objects to that they
  appear to be part of the video scene.
• Showing a small red ball in the most recent
  stereo video frame at the position of the gripper
  calculated locally using the command data from
  master arm.
• Overlaying requires a one-to-one mapping of
  remote and virtual world coordinate spaces using
  a camera model.
• We use the weak-perspective camera model.

80
Augmented Reality Camera Identification

• Using a camera model requires the identification
  of its projection matrix.
• Two projection matrices are needed for left and
  right images for a stereo projection.
• A 3D frame of reference serves as affine basis
  for all other points in the scene.
• This affine relationship between frame of
  reference and other points remains invariant in
  the projected points.

81
Augmented Reality Camera Identification

• IdentifyCamera component is designed to help
  identify both cameras at the system
  initialization as well as when required.

Reference Frame
82
Augmented Reality Surfaces, HAL, Page Flipping

• Microsoft DirectX is a set of highly optimized
  application programming interfaces (APIs) for
  developing high- performance 2D and 3D graphics
  (or multimedia) applications.
• A DirectX surface can be thought of a piece of
  paper that you can draw on. Provides access to
  pixels data.
• HAL (Hardware Abstraction Layer) provides a
  common set of graphics functions on all hardware
  devices.
• Primary surface is the current video buffer. We
  write our next frame data to off-screen secondary
  surface. In one instruction, graphics device
  flips the addresses of both surfaces sending the
  off-screen to output surface -- Page Flipping.

83
Augmented Reality Component Framework

• On the server side, no new component is added for
  the AR application. However server side requires
  setting up cameras, placement and removal of
  reference frame, etc.
• Client side has the following components
• StereoSocketClient component
• IdentifyCamera component
• RobotModel component
• DXInterface component

84
Augmented Reality StereoSocketClient Component

• A multi-threaded component initialized by client
  AR application to
• provide necessary un-blocking socket interface to
  vision server on the remote side by connecting
  and receiving data through a dedicated thread.
• extract single as well as stereo images from
  binary video data stream being sent from vision
  server.
• synchronize left and right images while providing
  stereo frames.
• Invokes an event when a new stereo frame is
  received from the server.

85
Augmented Reality StereoSocketClient Component
86
Augmented Reality RobotModel Component

• Acts as a passive proxy of PUMA robot on client
  side.
• Provides updated gripper and joint positions in
  Cartesian space through PUMA direct and inverse
  geometric models and respectively.
• IDecisionServer cannot be used because it is an
  active proxy of PUMA which does not allow
  manipulating the position of robot joints
  independent of PUMA.

87
Augmented Reality DXInterface Component

• Central component of AR framework.
• Runs AR and visualization business in separate
  threads.
• Handles several tasks such as
• Synchronization of real and virtual data
• Projection on video surface
• Augmentation of real video
• Page Flipping for HMD stereo visualization

88
Augmented Reality DXInterface Component
89
Augmented Reality Complete System
90
Augmented Reality Augmenting Video
Augmented Ball
Augmented Ball
Augmented Video
Normal Video
91
Conclusions

• Real-time control of telerobots in the presence
  of time delays and data loss is a dynamic
  research area.
• Efficient teleoperation by the operator requires
  the availability of force and visual feedbacks
  which, over a LAN, can only be attained through
  multi-streaming the real-time data.
• This work uses .NET based distributed components
  for the development of a reliable telerobotic
  scheme that offers multi-streaming the real-time
  data through extremely fast network connections
  in a multithreaded environment.

92
Contributions

• A highly optimized stereo video client-server
  framework is designed and developed using Visual
  C and Visual C.NET programming languages.
• With this framework we are able to achieve an
  excellent video transfer rate of 18 stereo frames
  per second over KFUPM LAN.
• Different output techniques for stereo video are
  implemented and performance evaluated like
  eye-shuttering glasses, HMD page flipping.

93
Contributions

• A component based multi-threaded distributed
  framework for telerobotics is designed,
  implemented and performance evaluated to study
  the effects of multi-threading on real-time
  telerobotics.
• This scheme has significantly reduced the network
  delays in a given telerobotic scenario while
  providing a very reliable connection between
  client and server sides.

94
Contributions

• Different geometric working frames are provided
  for the operator to enhance his maneuverability
  in the remote environment.
• Force feedback is deployed on the client side as
  a mean to enhance the tele-presence of the
  operator tele-manipulating the slave arm.
• Computer vision techniques are explored to create
  AR (augmented reality) on the client side by
  merging the virtual data with the real video
  stream from the remote side.
• The use of AR has helped in decreasing the
  network delays by reducing the requirement for
  fresh video data.

95
Future Research Directions

• Implementing hierarchical supervisory control in
  the developed telerobotic framework. This will
  allow repeatability of simple tasks using
  impedance control.
• Incorporation of complex geometrical shapes in
  the real video in order to provide even richer
  information to the client side.
• Studying the affects of hyper-threading on a
  multi-threaded telerobotic framework.
• Comparison of the projection accuracies of
  different camera models while augmenting the real
  data.
• Analysis and design of a 6 d.o.f. (3 d.o.f. force
  feedback) master
• arm being developed at KFUPM in COE department.

96
Thank You
Questions are welcome