Augmented Reality Applications

1
Augmented Reality Applications

• Cindy Robertson
• CS 7497 Spring 2000

2
Medical Applications

• Ultrasound-guided needle biopsies
• Visualization for laparoscopic surgery
• Image guided surgery
• Computer assisted dental implant surgery

3
Ultrasound Guided Needle Biopsies

• UNC Ultrasound Research
• Video see-through HMD
• Combines computer-generated ultrasound imagery
  with the live video image of a patient
• Can be used in obstetrics, diagnostic procedures
  (needle-guided biopsies), cardiology, etc.
• Replaces open surgical intervention

4
Ultrasound Guided Needle Biopsies (cont.)

• Ultrasound guidance is difficult to perform -
  need really good hand-eye coordination and 3D
  visualizations skills to guide the biopsy needle
  to the target tissue area with the aid of
  ultrasound imagery
• Both HMD video and ultrasound video are captured

5
Ultrasound Guided Needle Biopsies (cont.)

• The camera video is played in the background

• Ultrasound video images are transferred into
  texture memory and displayed on polygons emitted
  by the ultrasound probe inside a synthetic
  opening within the scanned patient

6
Ultrasound Guided Needle Biopsies (cont.)

• SIGGRAPH 96 video
• Despite efforts at registration, they still do
  not achieve the high precision necessary for
  performance in surgical procedures
• They are investigating alternative tracking
  systems

7
Laparoscopic Visualization Research

• UNC
• Laparoscopy suffers from
• limited field of view
• lack of good hand-eye coordination
• 2D imagery displays lack depth information
• Use AR to automatically extract depth information
  and geometry structures
• Displays operating field from surgeons point of
  view

• Physician wont have to physically adjust
  endoscope to see different views

8
Image Guided Surgery

• Collaboration between MIT AI lab and Brigham and
  Womens Surgical Planning lab

• Will enable surgeons to visualize internal
  structures through an automated overlay of 3D
  reconstructions of internal anatomy on top of
  live video views of a patient

9
Image Guided Surgery (cont.)

• Construct 3D models using scans such as MR and
  CT.
• Segment data to distinguish between tissue types
  and surface render the data

10
Image Guided Surgery (cont.)

• Set up in Operating Room consists of
• portable cart containing a Sun UltraSPARC
• hardware to drive the laser scanner
• Flashpoint tracking system

11
Image Guided Surgery (cont.)

• Laser scanner used to collect 3D data of
  patients scalp surface as positioned on
  operating table to register the patient to the
  segmented MR skin
• Coordinate frames must be transformed
• Enhances visualization because the surgeon has
  x-ray vision
• Medical instruments are tracked

12
Image Guided Surgery (cont.)

• Impact of Surgical Navigation System
• used for over 200 neurosurgical cases
• used 1-2 times/week
• achieves high positional accuracy
• monetary saving are huge
• makes surgeons more confident

13
Computer Assisted Dental Implant Surgery

• Artma Biomedical
• Dental headpiece tracked by 3D sensors
• Dentist navigates the drill relative to planned
  implant position visualized in HMD

14
Entertainment Applications

• Weather reports
• Special effects
• Advertisements in broadcast images
• Games

15
Weather Reports

• Reporter standing in front of a blue or green
  screen
• Real image augmented with computer generated maps
  using chroma-keying

16
Special Effects

• Digital compositing creates illusions
• May not be considered augmented reality because
  not generated in real-time

17
Advertisements in Broadcast Images

• Princeton Electronic Billboard
• System allows broadcasters to insert
  advertisements into specific areas of broadcast
  images

18
Games AR2 Hockey

• Augmented Reality Air Hockey game
• Uses virtual puck
• Video VRAIS 98

19
Military Training Applications

• Displays in cockpits
• Used in military aircraft
• Presents information to the pilot on
• the windshield
• the visor of their flight helmet

20
Engineering Design

• Joint design reviews of complex devices in
  physically separated environments and real-time
  interaction with elements of a design
• Architectural Anatomy
• ARC Augmented Reality for Construction

21
Design Reviews

• Clients and designers each have a conference room
  equipped with an AR display
• Designers show their mocked up design in the
  clients conference room in 3D
• Clients walk around in display, point at
  prototype to highlight sections to discuss
• Highlighted areas reflected on real model in
  augmented display that designers are using
• Clients can interact with design and make changes

22
Architectural Anatomy

• Columbia University- Computer Graphics and User
  Interfaces Lab
• Overlays a graphical representation of portions
  of a buildings structural systems over users
  view of room

• Virtual image shows outlines of concrete joists,
  beams, and columns

23
ARC - Augmented Reality for Construction

• Columbia University - Computer Graphics and User
  Interfaces Lab
• Developed a testbed augmented reality system that
  addresses spaceframe construction
• Spaceframes are typically made from a large
  number of components of similar size and shape

24
ARC - Augmented Reality for Construction (cont.)

• Easy to assemble pieces in the wrong
  position-which if undetected could lead to
  structural failure
• System designed to guide workers through the
  assembly of a spaceframe structure, to ensure
  that each member is properly placed and fastened.
• Video

25
Robotics and Telerobotics Applications

• Telerobotic operators use visual images of a
  remote workspace to guide robots
• Operators can practice motions on virtual robots
  before they proceed with the real robot

26
Manufacturing, Maintenance, and Repair
Applications

• Mechanical Repair
• Repair Instruction
• Maintenance Assistance

27
Mechanical Maintenance and Repair

• User Interaction and Visualization group at the
  European Computer-Industry Research Centre

• System provides a variety of information
• names of parts
• describes their function
• present maintenance or manufacturing records

28
Repair Instruction

• Military development
• Wireless vest worn by personnel
• Attached to optical see-through display
• Wireless connection allows soldier to access
  repair manual and images of equipment

29
Repair Instruction (cont.)

• MIT Media Lab
• Distinctive markings allow for an objects 3D
  location to be derived by a wearable camera
• Machine transmits diagnostics to technicians
  wearable

30
Repair Instruction (cont.)

• Wearable determines problem, locates 3D position
  of object, overlays specific 3D real-time
  step-by-step guidelines for technician to follow

31
Consumer Design

• Interior design
• Remodeling your house
• Virtually trying on clothes
• Fashion design

32
Interior Design

• Computer monitor shows picture of room from
  viewpoint of camera.
• Select furniture over network from various
  manufactures who make models available in their
  databases

• 3D renderings of furniture appear on monitor
  along with view of room.

33
Touring Machine (MARS)

• Columbia University - Computer Graphics and User
  Interfaces Lab
• Combines Augmented Reality and and mobile
  computing
• MARS unit acts as an campus information system
• Assists user in finding places and allows user to
  query information about items of interest, like
  buildings, statues, etc.
• Video

34
Interaction with the Real World

• Sony Computer Science Lab
• Ways to interact with the real world
• Example library setting
• Video UIST 95

35
Audio

• Beth Mynatts Audio Aura
• Uses auditory cues triggered by movements in
  physical space to provide awareness of email or a
  colleagues presence and activities
• Is this really augmented reality?