PPT – MULTISTREAM REAL-TIME CONTROL OF A DISTRIBUTED TELEROBOTIC SYSTEM PowerPoint presentation | free to download - id: 697117-NjVkZ


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation



multistream real-time control of a distributed telerobotic system asif aqbal – PowerPoint PPT presentation

Number of Views:27
Avg rating:3.0/5.0
Date added: 19 November 2019
Slides: 97
Provided by: AsifI4


Write a Comment
User Comments (0)
Transcript and Presenter's Notes


Asif Aqbal
  • Introduction
  • Background
  • Status Of The Problem
  • Literature Review
  • Thesis Objectives
  • Video Client-Server Framework
  • Distributed Telerobotic Framework
  • Augmented Reality
  • Conclusions
  • Thesis Contributions
  • Future Research Directions

  • Telerobotics humans to extend their manipulative
    skills over a distance, extend eye-hand motion
  • Telerobotic applications
  • Scaled-down nano-scale, micro-surgery,
  • Hazardous nuclear decommissioning inspection,
    fire fighting, disposal of dangerous objects,
    minefield clearance, operation in harsh
    environments, unmanned, underwater, ice, desert,
  • Safety rescue,
  • Security surveillance, reconnaissance,
  • Unmanned oil platform inspection, repair,
  • Teaching, training, and entertainment.

Introduction (cont.)
  • Minefield clearance, unmanned underwater
    inspection, and search rescue.
  • Those where humans adversely affect the
    environment such as medical applications and
    clean-room operations.
  • Those which are impossible for humans to be
    situated in such as deep space and nanorobotics.

Introduction (cont.)
  • Extending eye-hand motion coordination using
  • In natural eye-hand motion coordination, operator
    sees his hand and react accordingly.
  • In telerobotics
  • Operator holds a master arm to dictate his hand
  • Motion is transmitted to a remote slave arm and
    reproduced (replica),
  • Operator wears a head-mounted display (HMD) to
    see in 3D the effects of his motion on the remote
  • Operator does not see his hand (HMD) nor the
    master arm, his hand is logically mapped to the
    remote tool,
  • Operator logically acts on the remote tool seen
    through the HMD.
  • A step further, stereo vision provides 3D views
    of slave scene to the operator. It also provides
    a metric to calculate 3D positions and
    orientations of the physical object present in
    the remote world.

  • Computer Systems Engineering integrating
    computer software and architecture, control, and
  • Software real-time client-server, RT-OS,
    distributed-components, multi-threading, thread
  • Architecture/compiler parallel streaming buses,
    parallel processing, interfacing, stereo-vision,
  • Control stability vs delay, manual, supervisory,
    automatic, position/force, shared control,
  • Mechanical light, ergonomic, wire-based
    transmission, backdrivable, dexterous, force
    fidelity, maintainable.

  • Visual feedback from the remote side plays an
    important role in increasing the efficiency of
    the operator tele-manipulating the robot.
  • Supervisory Control as defined by Sheridan1986
    Supervisory control means that one or more human
    operators are intermittently programming and
    continually receiving information from a computer
    that itself closes an autonomous control loop
    through artificial effectors to the controlled
    process or task environment.

Background (cont.)
  • Specifically a Telerobotic system consists of
  • A master arm connected to a client program that
    monitors the operator motion and interacts with a
    slave station to ensure that the slave arm motion
    is a replica of operator motion.
  • A slave arm connected to a server program that is
    capable of converting a relative operator motion
    into an incremental slave motion.

Background (cont.)
  • A two-way logical communication link to transfer
    commands from client to the server through a
    Computer Network and to convey LAN
  • different kinds of feedback, e.g., video, force
    etc., back to the client site.

Background (cont.)
  • A Telepresence system is one which displays high
    quality information from the remote world, visual
    or otherwise, in such a natural way that the
    operator feels physically present at the remote
  • Virtual Reality (VR) is the interactive
    simulation of a real or imagined environment that
    can be experienced visually or otherwise in the
    three dimensions of width, height, and depth.

Background (cont.)
  • Augmented Reality (AR) superimposes graphics and
    virtual objects to a stereo view of slave site in
    an attempt to provide the operator, an
    anticipated view of his actions ahead of time
    which is useful to overcome overall system
  • Or it is a variation of VR in a sense that AR
    supplements reality while running the interactive
    VR simulation.

Background (cont.)
  • Kinesthesis are the sensations derived from
    muscles, tendons and joints and stimulated by
    movement and tension.
  • A haptic device is one that displays the
    kinesthesis to a human hand or arm.

Status Of The Problem
  • Responsiveness, real-time, and quality-of-service
    are needed to provide high-quality interactions
    between operator and slave arm.
  • Issues in remote sensing of the slave environment
    e.g., stereo video, force sensing, sound etc.
  • Faster acquisition of video data, packing,
    zero-copy, transmission, etc.
  • Camera identification problems.

Status Of The Problem
  • Fair allocation of processor usage to multiple
    threads for control, sensors, GUI, and windows
  • Force sensing using high precision sensors,
    passive and active compliance, force control,
    impedance control, and mixed position-force

Status Of The Problem
  • Time delay problems.
  • Processing, copying video data, network delays.
  • Internet and LANs dont provide the Q.o.S.
  • Manual control of the remote manipulator is
    impeded if there is a time delay between command
    and feedback. Operator switches to Move-n-Wait
  • Can drive continuous closed loop systems to

Literature Review
  • Supervisory Control
  • Shridan1989
  • Mathew1995
  • Christian1996
  • Chen1997
  • Ren2000
  • Stereo Vision and Augmented Reality
  • Dave1994
  • Kuna1995
  • Milgram1995
  • Spyros1999
  • Network Telerobotics
  • Paolucci1996
  • Wayne1996
  • Kazuhiro1996
  • Yeuk1999
  • Yeuk2000

Thesis Objectives
  • Advanced Telerobotic Software
  • To develop a fully Distributed Object Oriented
    Framework using the most advanced software
    authoring tools (.NET and SOAP technologies) for
    real-time control of Robot-Server-Client
    interactions including upgrading of previously
    implemented kinematic and inverse-kinematic
  • Investigation of impact of .NET component based
    approach on network time delays while deploying a
    multi-stream, multi-threaded distributed
    application having command, force and video
    streams at the same time.

Thesis Objectives
  • Kinesthetic Mapping of Master-Slave Interaction
  • Implementing geometric working frames, like tool
    and robot frames to improve the mapping between
    master and slave arms.
  • Implementing a scalable mapping between master
    and slave spaces that can be controlled by the
    operator in real-time.
  • Providing the operator with force-feedback
    display by using real-time stream of force

Thesis Objectives
  • Computer Vision
  • Development of an optimized client-server
    framework for grabbing, transmission and display
    of 3D stereo data over a LAN. 3D effects will be
    produced using different methods like
    sync-doubling, page-flipping etc.
  • Investigation of Augmented Reality as part of a
    strategy to reduce network delays in a
    telerobotic environment by using simple pointer
    in the stereo image at the client side.

Video Client-Server Framework
  • The provision of stereo video on the client side
    imposes severe requirements in terms of bandwidth
    to transfer real-time stream of video data in a
    telerobotic environment.
  • It requires the use of advanced technologies like
    DirectX and Windows Sockets to accomplish the
    capturing and relaying of video data over a LAN.
  • Commercially available software like Microsoft
    NetMeeting are optimized for a low band-width
    network like internet so they show too poor
    display resolution to be used for stereo vision
    in a telerobotic setup.

Video Client-Server Framework
  • Development of a highly optimized client-server
    framework for grabbing and relaying of a stereo
    video stream becomes inevitable for an efficient
    telerobotic framework.
  • This framework must accomplish following tasks on
    the server side
  • Capture or grab stereo images from two cameras at
    the slave side simultaneously.
  • Establish a reliable client-server connection
    over a LAN, the slave side being the vision
  • Upon requests from the client send this stereo
    frame comprising of two pictures to the client
    through windows sockets.

Video Client-Server Framework
  • On the client side it should provide the
    following functionality
  • Establish a highly optimized fast graphic display
    system to show the pictures received from the
  • Detect and establish the connection with server.
  • Display the pictures arrived from the server and
    continue in a loop each time asking a new stereo
    frame from the server.
  • Allow the viewer to adjust the alignment of the
    pictures on the output device, whatever it is, to
    compensate for the misalignment and
    non-linearities present in the stereo camera
    setup at server side.

Video Client-Server Framework
  • A client-server framework fulfilling the above
    defined requirements is developed using the
    advanced software development tools like
    Microsoft Visual C and Microsoft DirectX.
  • Microsoft DirectX provides COM based interfaces
    for various graphics related functionalities.
    DirectShow is one of these services. DirectShow,
    further, provides efficient interfaces for the
    capturing and playback of video data.

Video Client-Server Framework
  • We can use network services and send/receive data
    over a network using windows sockets. The stereo
    video setup uses synchronous windows sockets as
    an interface between vision server and client.
  • Two different schemes were implemented to
    transfer the video data. The schemes differ in
    the usage of multiple threads on the server side
    as well as some optimization steps to reduce the
    network traffic for the transfer of the video
  • A general overview of the image grabbing and
    displaying system is given before the detailed
    description of the above scheme.

Video Client-Server Framework
  • We use a component of DirectShow named
    SampleGrabber to capture video frames coming
    through a stream from a stereo camera setup. A
    block diagram of the scheme used at the server
    side to grab stereo frames is shown below

Video Client-Server Framework
  • In order to show the received pictures from the
    server, we need to use GDI (Graphics Device
    Interface). A block diagram of the client side
    scheme to display the video is shown below

Video Client-Server Framework (Single Buffer,
Serialized Transfer)
Video Client-Server Framework Double Buffer,
De-Serialized Transfer
  • In this scheme, we try to optimize the transfer
    of video data over the LAN by using thread
    manipulation on the server.
  • Thread overlapping among capture and sending
    thread is achieved using double buffers on the
    server side.
  • It is ensured that the thread responsible for
    sending the video data over the LAN will not wait
    after receiving a picture request from the client.

Video Client-Server Framework Double Buffer,
De-Serialized Transfer
Video Client-Server Framework Double Buffer,
De-Serialized Transfer
  • This approach enables us to send higher number of
    stereo frames over the same LAN and hardware.
  • The only overhead is the allocation of extra
    buffer in the server DRAM which not a real
    problem with available systems containing large

Video Client-Server Framework 3D Visualization
  • There can be different methods to produce 3D
    effects on the client side once we have stereo
    images of the remote scene.
  • Similarly different hardware device such as
    eye-shuttering glasses, HMD (Head Mounted
    Display) are used to show the images to the user.
  • We have used following two methods for stereo
    image production on client side
  • Sync-Doubling
  • Page Flipping

Video Client-Server Framework Sync-Doubling
  • Left and right eye images are arranged in an up
    and down way on the computer screen.
  • A sync-doubler sits between the display output
    from the PC and the monitor to insert an
    additional frame v-sync between the left and
    right frames (i.e. the top and bottom frames).
  • This will allow the left and right eye images to
    appear in an interlaced pattern on screen.
  • Using the frame v-sync as the shutter alternating
    sync allows us to synchronically transmit the
    right and left frames to respective left and
    right eyes, thus creating a three-dimensional

Video Client-Server Framework Sync-Doubling
Video Client-Server Framework Page Flipping
  • Page-flipping means alternately showing the left
    and right eye images on the screen.
  • Combining the 3D shuttering glasses with this
    type of 3D presentation requires the application
    of frame v-sync as the shutter alternating sync
    to create a 3D image.
  • HMD can also be used in a way that two different
    images are sent on two different LCD screens of
    the HMD. The user sees the different image for
    both eyes thus feeling the depth of the scene.
    DirectX can be used to flip both the images

Video Client-Server Framework Performance
  • Different experiments were conducted to test the
    visual quality of the client-server setup as well
    as find the time delays and other measures of the
    video data.
  • The specifications of the stereo frame are as
  • Height of each picture 288 pixels
  • Width of each picture 360 pixels
  • Size 304 KB (311040 Bytes) per picture
  • 608 KB (622080 Bytes) per stereo
  • Each stereo frame is of size 0.6 MB and requires
    a bandwidth of about 5Mbps/Frame on the LAN. This
    simple calculation shows the limitation of the
    100 Mbps LAN to transfer only 20 fps at the
    highest possible transfer rate.

Video Client-Server Framework Performance
  • Copying from SampleGrabber to DRAM
  • Case 1 Copy times on server Single Force
  • 300 stereo frames
  • Mean value 24.025 ms
  • 95 CI between 23.29 ms and 24.75 ms.

Video Client-Server Framework Performance
  • Copying from SampleGrabber to DRAM
  • Case 2 Copy times on server - Two Threads
  • 300 stereo frames
  • Mean value 60.48 ms
  • 95 CI between 8 ms and 150 ms.

Video Client-Server Framework Performance
  • Copying from SampleGrabber to DRAM
  • Case 3 Copy times on server with Force transfer
    over LAN
  • 300 stereo frames
  • Mean value 33.46 ms
  • 9.43 ms additional for adding network transport

Video Client-Server Framework Performance
  • Transferring over the LAN
  • Case 1 Single Buffer, Serialized Transfer
  • 300 stereo frames
  • Mean value 86.1 ms
  • 11.61 stereo frames/second.

Video Client-Server Framework Performance
  • Transferring over the LAN
  • Case 2 Double Buffer, De-Serialized Transfer
  • 60,000 stereo frames
  • Mean value 58.94 ms
  • 17 stereo frames/second.
  • 90 CI between 56.0 and 64.8 ms.

Video Client-Server Framework Results Summary
Scheme Cameras to Server DRAM (ms) Server to Client (ms) Frames Per Second
Single Buffer, Serialized 24.025 86.1 11.61
Double Buffer, De-serialized 24.025 58.94 17
  • Housheng et. al.2001 reported a transfer rate
    of 9-12 fps for a compressed single image of size
    200X150 pixels over a LAN. While our scheme
    transfers 17-18 uncompressed stereo fps of size
    360X288 pixels each.
  • Network bandwidth is near saturated with 18 fps.

A Multi-threaded Distributed Telerobotic Framework
  • Distributed application programming is one of the
    different schemes to establish a reliable
    connection between master and slave arms.
  • Different items are realized as software
    components and then these components communicate
    with each other using distributed components
  • Object Oriented Approach
  • Software reusability
  • Easy extensibility
  • One time debugging
  • Multi-user environment
  • Data encapsulation

A Multi-threaded Distributed Telerobotic Framework
  • By using the distributed programming, network
    protocol issues can be avoided. The distributed
    framework itself takes care of all the network
    resources and binary data transfer over the
  • Previously DCOM (Distributed Component Object
    Model) based components have been used in
    telerobotics by Yeuk et. al.
  • .NET components are more advanced than COM based
    components and offer complete support of .NET
    framework including .NET Remoting and SOAP
  • Several components are developed on server as
    well as client side and will be explained briefly.

A Multi-threaded Distributed Telerobotic
Framework Server Side Components
  • PUMA Component
  • Force (Sensor) Component
  • DecisionServer Component

A Multi-threaded Distributed Telerobotic
Framework Server Side Components
  • PUMA Component
  • PUMA component is the heart of the distributed
    framework as this deals with the commands being
    sent to robot and the response of the robot.
  • This component acts as a software proxy of the
  • Commands are issued to the component as they are
    being issued to the robot. Whenever robot, under
    the action of these commands, changes its states,
    the component updates itself automatically to
    reflet these changes.

A Multi-threaded Distributed Telerobotic
Framework PUMA Component
A Multi-threaded Distributed Telerobotic
Framework PUMA Component
  • PUMA offeres public methods like
  • InitializeRobot, to initialize the robot to a
    specific position or orientation.
  • MoveIncremental, to move the robot incrementally
    in either joint or cartesian space. Overloaded.
  • MoveAbs, moves the robot to the absolute
    position supplied as an argument in joint or
    cartesian space. Overloaded.

A Multi-threaded Distributed Telerobotic
Framework PUMA Component
  • The component also provides access to many useful
    values through public properties like
    CurrentOrienationMatrix, CurrentPositionVecotr,
    CurrentJointAngles, RS-232Settings,
    CurrentWorkingFrameMode, etc.
  • In addition, the component invokes different
    events conveying the changes in robot states like
    RobotMoved, DataReceived, StatusChanged,
    ErrorOccured, etc.
  • Robot status is described by 6 discrete states,
    the change in which is notified by StatusChanged

A Multi-threaded Distributed Telerobotic
Framework PUMA move in joint space
Any No will cause a FALSE return.
A Multi-threaded Distributed Telerobotic
Framework PUMA move in Cartesian space
Any No will cause a FALSE return.
A Multi-threaded Distributed Telerobotic
Framework Server Side Components
  • Force Sensor Component
  • This component takes care of the force and/or
    position sensing operations carried out on the
    server as well as client sides.
  • Force sensing operation is done in a dedicated
    read thread, the priority of which can be
    adjusted during runtime to allow for the better
    management of CPU usage.
  • Due to multi-threading, the Force Sensor
    component supports both the reading and writing
    operations simultaneously.

A Multi-threaded Distributed Telerobotic
Framework Force Sensor Component
A Multi-threaded Distributed Telerobotic
Framework Force Sensor Component
  • Force Component offers public methods like
  • StartReading, creates a separate reading thread
    with default thread priority and sampling
  • StopReading, stops and aborts the force reading
  • The component exposes public properties like
  • SensorThreadPriority, to set sensor thread
  • TimerValue, to set the sampling interval.
  • ThresholdValue, sets the minimum change in force
    value required to invoke the ForceDataRead event.

A Multi-threaded Distributed Telerobotic
Framework Server Side Components
  • DecisionServer Component
  • An inherited component possessing the
    functionalities of both PUMA and Force Sensor
    components and adding more to it.
  • To remotely access the public members of PUMA and
    Force Sensor components using IDecisionServer
    interface in order to realize a truly distributed
  • To implement a supervisory control and learning
    mode in a higher abstraction layer during
    possible future work.
  • To Implement different modifiers to the commands
    coming from the client, e.g. workspace
    scalability function.

A Multi-threaded Distributed Telerobotic
Framework DecisionServer Component
A Multi-threaded Distributed Telerobotic
Framework Server Side Components
  • Server Side Interfaces and .NET Remoting
  • An interface is a set of public methods and
    definitions. It serves as a contract for any
    component that implements this interface.
  • All the server side components must implement the
    interface IDecisionServer which in turn is
    inherited from IForce and IProxyRobot.
  • This scheme is required because .NET Remoting (a
    component of .NET framework) requires the
    interfaces on client side to provide references
    to the server side objects.

A Multi-threaded Distributed Telerobotic
Framework Client Side Components
  • IDecisionServer Interface
  • MasterArm Component

A Multi-threaded Distributed Telerobotic
Framework Client Side Components
  • IDecisionServer Interface
  • In the start carries only an un-referenced
    interface to DecisionServer component.
  • After establishing a network connection with the
    server, the client gets reference to server side
    instance of DecisionServer component through
    IDecisionServer interface.
  • Client side can access the methods and properties
    of DecisionServer similar to invoking them on a
    local component.

A Multi-threaded Distributed Telerobotic
Framework Client Side Components
  • MasterArm Component
  • Implements all the functionality required to
    interact with a force feedback master arm in
  • After initialization it has an active instance
    of Force component with two active threads, one
    each for reading and writing position/force
  • It also implements the local force feedback to
    facilitate the operator by reducing the amount of
    force required to manipulate the master arm
    similar to a power steering mode.

A Multi-threaded Distributed Telerobotic
Framework MasterArm Component
  • Local force feedback uses a second order model
    for minimizing the force applied by the operator.
  • In order to estimate the force, the component
    maintains a record of all the force data read for
    a certain number of samples (history) along with
    the record of the system time.
  • Then it evaluates the velocity and acceleration
    of the master arm at each sampling instant and
    stores them in a circular buffer.
  • This information is used to calculate the force
    proportional to what the operator is applying
    which is then fed back to the master arm.

A Multi-threaded Distributed Telerobotic
Framework MasterArm Component
A Multi-threaded Distributed Telerobotic
Framework MasterArm Component
  • Some of the public methods exposed by MasterArm
    component are
  • StartReading, starts reading the position data
    from the master arm. Inherited from Force
  • StopReading, stops reading position data.
  • WriteForceData, writes supplied force data to
    output channels for force output device.
  • The component exposes public properties like
  • Incremental Orientation Matrix, provides the
    incremental orientation matrix when new position
    data is read.
  • Incremental Position Vector, provides the
    incremental position vector.

A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme
  • To receive the DecisionServer events on client
    side for further processing, DecisionServer needs
    the client assembly to be visible.
  • Violates the object oriented philosophy.
  • Restricts the deployment to multi-user
  • Shim classes are used as intermediate agents to
    forward server side events to the client side.
    They are compiled in a common library component
    visible to both server and client.

A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme
Server Side
A Multi-threaded Distributed Telerobotic
Framework Integrated Component Scheme
Client Side
A Multi-threaded Distributed Telerobotic System
  • With all the client and server components, we now
    proceed to a full-blown distributed telerobotic
  • This is multithreaded because more than one
    thread are running on the client and server for
    different tasks like image grabbing,
    reading/writing force sensors, sending control
    signals to robot, etc.
  • Transferring real-time streams of force, command
    and stereo video over LAN makes it a
    multi-streaming environment.
  • The logic of the framework is distributed among
    different hardware and software components.

A Multi-threaded Distributed Telerobotic System
A Multi-threaded Distributed Telerobotic System
Client GUI
A Multi-threaded Distributed Telerobotic System
Performance Evaluation
  • LAN Bandwidth 100 Mbps
  • Client and Server located in the same lab
  • Client and Server 2.0 Ghz P-IV machines.
  • 1 GB DRAM
  • Accelerated Graphics Board on Client
  • Each force packet 6 double values
  • 48 bytes

A Multi-threaded Distributed Telerobotic System
Performance Evaluation
  • Force transfer only
  • 3000 force packets.
  • Mean inter-arrival time 0.679 ms
  • 90 CI between 0.59 to 0.92 ms.
  • Worst case inter-arrival 825 ms.

A Multi-threaded Distributed Telerobotic System
Performance Evaluation
  • Force and video streams
  • 3000 force packets.
  • Mean inter-arrival time 1.08 ms
  • An addition of 0.4 ms.
  • 90 CI between 0.5 and 3.9 ms.
  • Worst case inter-arrival 789.74 ms.
  • During the transfer of video data
  • 3710 force packets.
  • Mean inter-arrival time 3.9 ms
  • 90 CI between 0.5 and 13 ms.

A Multi-threaded Distributed Telerobotic System
Performance Evaluation
A Multi-threaded Distributed Telerobotic System
Performance Evaluation
  • A magnified plot of inter-arrival times in the
    presence of force, video and command streams.

A Multi-threaded Distributed Telerobotic System
A comparison
  • Teresa1999 developed JAVA and VRML based
    telerobotic system and reported a image
    acquisition time of 1s for one single frame of 16
    bit depth. Our DirectShow based system reports a
    24 ms stereo image acquisition time in a
    telerobotic system.
  • Al-Harthy2001 implemented client-server
    framework takes around 50ms to transfer a command
    signal (48 bytes) from client to robot. In our
    case a similar packet (48 bytes) takes from 0.7
    to 1.1 ms due to the efficient utilization of raw
    network resources by .NET Remoting.

A Multi-threaded Distributed Telerobotic System
  • Lifting a 100 gram weight

A Multi-threaded Distributed Telerobotic System
  • Pressing and maintaining the contact on a spring

Augmented Reality
  • The basic idea of an AR (augmented reality)
    reality system is to mix the real and virtual
    information in order to provide an augmented view
    of the remote scene that provides more
    information than a simple video could offer.
  • AR can be used as an effective way to overcome
    the effects of time delays in a telerobotic
  • The information added locally must fit seamlessly
    into the remote real data so as to avoid any
    perplexities for the teleoperator.

Augmented Reality Work Strategy
  • To introduce non-existent objects to that they
    appear to be part of the video scene.
  • Showing a small red ball in the most recent
    stereo video frame at the position of the gripper
    calculated locally using the command data from
    master arm.
  • Overlaying requires a one-to-one mapping of
    remote and virtual world coordinate spaces using
    a camera model.
  • We use the weak-perspective camera model.

Augmented Reality Camera Identification
  • Using a camera model requires the identification
    of its projection matrix.
  • Two projection matrices are needed for left and
    right images for a stereo projection.
  • A 3D frame of reference serves as affine basis
    for all other points in the scene.
  • This affine relationship between frame of
    reference and other points remains invariant in
    the projected points.

Augmented Reality Camera Identification
  • IdentifyCamera component is designed to help
    identify both cameras at the system
    initialization as well as when required.

Reference Frame
Augmented Reality Surfaces, HAL, Page Flipping
  • Microsoft DirectX is a set of highly optimized
    application programming interfaces (APIs) for
    developing high- performance 2D and 3D graphics
    (or multimedia) applications.
  • A DirectX surface can be thought of a piece of
    paper that you can draw on. Provides access to
    pixels data.
  • HAL (Hardware Abstraction Layer) provides a
    common set of graphics functions on all hardware
  • Primary surface is the current video buffer. We
    write our next frame data to off-screen secondary
    surface. In one instruction, graphics device
    flips the addresses of both surfaces sending the
    off-screen to output surface -- Page Flipping.

Augmented Reality Component Framework
  • On the server side, no new component is added for
    the AR application. However server side requires
    setting up cameras, placement and removal of
    reference frame, etc.
  • Client side has the following components
  • StereoSocketClient component
  • IdentifyCamera component
  • RobotModel component
  • DXInterface component

Augmented Reality StereoSocketClient Component
  • A multi-threaded component initialized by client
    AR application to
  • provide necessary un-blocking socket interface to
    vision server on the remote side by connecting
    and receiving data through a dedicated thread.
  • extract single as well as stereo images from
    binary video data stream being sent from vision
  • synchronize left and right images while providing
    stereo frames.
  • Invokes an event when a new stereo frame is
    received from the server.

Augmented Reality StereoSocketClient Component
Augmented Reality RobotModel Component
  • Acts as a passive proxy of PUMA robot on client
  • Provides updated gripper and joint positions in
    Cartesian space through PUMA direct and inverse
    geometric models and respectively.
  • IDecisionServer cannot be used because it is an
    active proxy of PUMA which does not allow
    manipulating the position of robot joints
    independent of PUMA.

Augmented Reality DXInterface Component
  • Central component of AR framework.
  • Runs AR and visualization business in separate
  • Handles several tasks such as
  • Synchronization of real and virtual data
  • Projection on video surface
  • Augmentation of real video
  • Page Flipping for HMD stereo visualization

Augmented Reality DXInterface Component
Augmented Reality Complete System
Augmented Reality Augmenting Video
Augmented Ball
Augmented Ball
Augmented Video
Normal Video
  • Real-time control of telerobots in the presence
    of time delays and data loss is a dynamic
    research area.
  • Efficient teleoperation by the operator requires
    the availability of force and visual feedbacks
    which, over a LAN, can only be attained through
    multi-streaming the real-time data.
  • This work uses .NET based distributed components
    for the development of a reliable telerobotic
    scheme that offers multi-streaming the real-time
    data through extremely fast network connections
    in a multithreaded environment.

  • A highly optimized stereo video client-server
    framework is designed and developed using Visual
    C and Visual C.NET programming languages.
  • With this framework we are able to achieve an
    excellent video transfer rate of 18 stereo frames
    per second over KFUPM LAN.
  • Different output techniques for stereo video are
    implemented and performance evaluated like
    eye-shuttering glasses, HMD page flipping.

  • A component based multi-threaded distributed
    framework for telerobotics is designed,
    implemented and performance evaluated to study
    the effects of multi-threading on real-time
  • This scheme has significantly reduced the network
    delays in a given telerobotic scenario while
    providing a very reliable connection between
    client and server sides.

  • Different geometric working frames are provided
    for the operator to enhance his maneuverability
    in the remote environment.
  • Force feedback is deployed on the client side as
    a mean to enhance the tele-presence of the
    operator tele-manipulating the slave arm.
  • Computer vision techniques are explored to create
    AR (augmented reality) on the client side by
    merging the virtual data with the real video
    stream from the remote side.
  • The use of AR has helped in decreasing the
    network delays by reducing the requirement for
    fresh video data.

Future Research Directions
  • Implementing hierarchical supervisory control in
    the developed telerobotic framework. This will
    allow repeatability of simple tasks using
    impedance control.
  • Incorporation of complex geometrical shapes in
    the real video in order to provide even richer
    information to the client side.
  • Studying the affects of hyper-threading on a
    multi-threaded telerobotic framework.
  • Comparison of the projection accuracies of
    different camera models while augmenting the real
  • Analysis and design of a 6 d.o.f. (3 d.o.f. force
    feedback) master
  • arm being developed at KFUPM in COE department.

Thank You
Questions are welcome