Trinetra Independence for the Blind and Visually Impaired

1
TrinetraIndependence for the Blind and Visually
Impaired
Phase 3 Implementation
2
HCI Rapid Prototyping Spring 06

• 16 students
• One full semester
• Phase 1 Concept Exploration
• Phase 2 Feasibility Reassessment
• Phase 3 Prototype Implementation

3
Instructors

• Daniel Siewiorek
• Asim Smailagic

4
Clients

• Priya Narasimhan
• ECE/CS Faculty
• Expertise embedded systems, distributed systems,
  fault tolerance, middleware
• Passionate about ensuring Trinetra leads to
  improvement in quality of life
• Dan Rossi
• Oracle system administrator
• Key driver and target user for Trinetra
• Key contributor in ensuring that the technology
  stays focused on the real needs and desires of a
  blind user

5
Presentation Order

• 1. HCI
• 2. Shopping
• 3. OCR
• 4. Transportation
• 5. Implementation Guidelines

6
Trinetra HCI Group

• Puneet Aggarwal
• Pablo Quinones
• Chris DeLeon
• Matt Lee
• Jim Lin

7
Visionary ScenarioLiving with Trinetra
8
Visionary Scenario 1 of 2

• Product Information (Shopping)
• Milk expiration date alert, medicine side effects
  check
• Object Identification (Shopping)
• Bottles in fridge, cereal boxes in pantry
• Navigation (Transportation)
• Knows how long before next shuttle will arrive
• Able to notify driver that hell be waiting
• Awareness of Location on Shuttle Ride
  (Transportation)
• Clients device privately calls out stops in
  earpiece
• Speaker on Bus Announces the stops

9
Visionary Scenario 2 of 2

• Sign Reading (OCR)
• Goes someplace new, can compare menu options to
  the posted special. Orders the special.
• Identifies bus stop, and which buses are on this
  route
• Product Information (Shopping)
• Identifies prices, safety warnings

10
Functional Capabilities

• Object Information
• Product prices, personal notes eg. serving size,
  expiration date, medication side effects
• Object Identification
• Cereal boxes on shelf, bottles in fridge
• Querying shuttle status
• Knows how long before next will arrive
• Able to notify driver that user will be waiting
• Awareness of location during shuttle ride
• Bus speaker calls out stops
• Awareness of menu options
• In a restaurant, user can compare menu options
  to posted specials
• Bus signs and information

Shopping
Transportation
Optical Character Recognition
11
Product Identification (Home)

• Goal
• Finding the carton of milk in fridge
• Interaction Physical browsing
• - Reach into fridge and touch
• - Audio from system names the product

12
Product Information (Store)

• Goal
• Find price and additional
• information like servings, calories,
• additional ingredients
• Interaction
• - user touches or holds product in hand
• - user presses button corresponding to a value
• - user hears the price, number of servings

13
Product Information (Home)

• Goal
• Find preparation/use instructions,
• expiration date
• Interaction
• - Points camera at package label
• - Voice reads out the label via earpiece
• - In future, information is available on the
  phone

14
Reading Signs

• Goal
• User aware of sign information
• Interaction
• - If sign placement known, user points camera
  and voice reads text into earpiece
• - Else, camera used to pan environment
• - User receives an audio cue if sign found -
  User presses a button on phone to hear text

15
Catching/Riding the Shuttle

• Goal
• Determine shuttle arrival
• Interaction
• - Launches Shuttle Helper app on phone
• - Prompts, Which shuttle stop?
• - Request Notification to the driver
• - Audio indication of ETA until next shuttle and
• shuttle interval
• - Notification when shuttle is near

16
User Artifacts

• Cell Phone (camera, text-to-speech)
• Earpiece
• Barcode Pencil
• RFID Reader IDBlue

17

• Interaction Architecture

18
Integrated Interaction Architecture
19
Interaction Architecture-2
20

• Overall Architecture

21
Local Database
Bluetooth
OCR
GPRS/EDGE
USB
Inventory
Picture of text from phone
OCR Server (ABBYY Software)
Cell Phone Functions

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Built-in camera

Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS GPRS
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
Laptop (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
22
Trinetra Shopping Group

• Paul Cheong
• Prateek Goenka
• Jun Han
• Rohan Kumar
• Patrick Lanigan
• Jim Lin

23
Local Database
Bluetooth
OCR
GPRS/EDGE
USB
Inventory
Picture of text from phone
OCR Server (ABBYY Software)
Cell Phone Functions

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Built-in camera

Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS GPRS
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
Laptop (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
24
Overview

• User interface
• Product information retrieval
• Local cache
• Local network
• Global network
• Input handlers and device drivers
• Bluetooth Device Discovery

25
User Interface
26
User Interface - 1
The user can choose to Connect to the scanning
devices, to clean the cache by selecting Purge
Cache, or to preload the cache by selecting Test
Cache After selecting Connect, a dialog appears
to notify the user that its connecting to devices
27
User Interface - 2
Once the user scan an item, it checks if the item
can be found in the local database first If the
item cannot be found in the local database, it
checks if the item can be found in the global
database
28
User Interface - 3
If the item is found, a dialog appears to notify
the user that the item is found, and a form that
contains the items name pops up If the item is
not found, a dialog appears to notify the user
that the item is not found, and a blank form pops
up
29
User Interface - 4
The user can edit the fields by selecting
Options, then select edit The user can
disconnect from the scanning devices by selecting
Disconnect
30
User Interaction
31
Information Retrieval
32
Local Cache
33
Cache Interface and Details

• Cache Interface
• Add Entry
• allows user to manually save product information
• can be used to save info when local/global
  network is not available
• Update Entry
• used to prevent redundant entries
• can be used by user to store additional
  information
• Retrieve Info
• used to retrieve previously saved product
  information

• Caching Details
• What to cache?
• Only cache global / static product information
  (ID, name, description)
• No localized / dynamic information (price, stock
  status, etc)
• Minimizes problems due to stale cache data
• How to cache it?
• Single binary file contains serialized cache
  objects
• Index stored in memory allows faster searching
• LRU cache replacement policy

34
Cache Entry and Cache Index

• A cache entry will have
• Unique ID
• 16 bytes
• unique produce idenfitier (barcode, tag ID)
• Product name
• 64 characters maximum
• Product description (notes)
• 512 characters maximum
• extended information (directions, warnings, etc)

• A cache index will
• Store metadata about cached items
• Allow efficient direct access to byte streams

TrinetraCacheIndexEntry
TStreamID StreamIDTBuf8lt16gt ItemIDTtime Created
Ttime ModifiedTtime Accessed
TrinetraCacheItem
TBuf8lt16gt ItemIDTBuflt64gt ItemNameTBuflt512gt ItemD
esc
35
Local Network
36
Components - Previous

• Bluetooth-to-Ethernet gateway (Parani100)
• Enables communication between Ethernet and
  Bluetooth-enabled devices
• Proxy server (Intrinsyc CerfCube)
• Responsible for filtering requests
• Queries database and returns replies
• MySQL database
• XML-RPC for communication

37
Proxy Server

• CerfCube Dropped
• Small Unix box with networking capabilities
• Couldnt find appropriate C ARM compiler
• Proxy developed on Andrew machine instead
• Proxy Server
• Phone connects to it over GPRS
• Listens to POST requests from phone
• Connects to Trinetra mySQL local inventory
  database to retrieve product information
• Light-weight and no concurrency

38
Experiments with BT-LAN Gateway
Procedures Observations / Inferences
Attempt to get IP address from CMU network on Parani100 Unable to obtain IP address after multiple tries
Tried enabling LAP (LAN Access Profile) to see if it made a difference to Parani100 No difference observed in IP address
Tried enabling Parani100 to be a DHCP client and wait for DHCP server No Response
Attempt to get IP address from Comcast provided internet on Parani100 Successfully obtained IP address
Installed Opera on Nokia 6620 Successful - In order to browse websites on cell-phone for testing
Installed gnubox on Nokia 6620 Successful - In order to directly ping Parani100 for testing
Successfully paired phone with Parani100 Unable to connect to internet
Decided to bypass Parani100 and connect directly to computer using bluetooth dongle Successful In order to test integrity of Parani100 Router
39
Experiments continued
Procedures Observations / Inferences
Initially unable to find bluetooth drivers for connecting bluetooth dongle with PC Successful Surfed multiple websites till adequate driver found
Installed BlueSoleil on laptop to interact correctly with Nokia 6620 the hope was to be able to connect directly to the local DB server bypassing Parani100 completely Phone and laptop paired successfully and the phone found the PC and vice versa, once again no contact from the phone side to the internet Opera displayed "contacting server" message but was unable to find anything after that
Connection between Nokia 6620 and PC severed several times due to unknown reason Due to unwitting movement of laptop by user
Contacted Parani100 suppliers for information regarding IP allocation using router No response
Properties page on Parani100 changed several times when refresh button clicked on browser Unknown
Downloaded several patches on phone to make it compatible with both Parani100 and BlueSoleil No Change in status
End Result Parani100 eliminated from
architecture
40
Components - Current

• Database and Proxy Server are integrated into a
  single box
• Queries are filtered at the application layer by
  the proxy before being processed by the MySQL
  database
• Replies are sent via GPRS/EDGE
• XML-RPC for communication

41
HTTP over Bluetooth

• Tried connecting to the www.upcdatabase.com
• using Bluetooth to share the internet connection
  from local server
• This would save user cost as it allows the user
  not to connect over the GPRS
• Did not work at the end so had to drop this

LAN
42
Global Network
43
Global Database

• www.upcdatabase.com/rpc
• Online UPC database provides access via XML-RPC
  communication protocol
• Same protocol as local network simplifies
  implementation on the phone
• Phone will use GPRS for Internet access and
  retrieve product information from
  www.upcdatabase.com via XML-RPC

44
System Integration
Request Broker
1.Local Cache
2. Local Network
3. Global Network
45
Input Handler

• Handles device service discovery
• Manages connections to multiple input devices
  (IDBlue, BaracodaPencil)
• Passes output from devices (barcodes, tag IDs) as
  input to request broker

46
Device Drivers

• Communicates with device over Bluetooth socket
• Implements device specific communication
  protocols
• Sets devices to correct operating modes
• Listens for and parses incoming data packets
• IDBlue Driver (shown right)

47
Trinetra Shopping(end)

• Paul Cheong
• Prateek Goenka
• Jun Han
• Rohan Kumar
• Patrick Lanigan
• Jim Lin

48
Trinetra OCR Group

• Tudor Dumitras
• Kunal Kapur
• Pablo Quinones
• Matt Lee

49
Local Database
Bluetooth
OCR
GPRS/EDGE
USB
Inventory
Picture of text from phone
OCR Server (ABBYY Software)
Cell Phone Functions

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Built-in camera

Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS GPRS
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
Laptop (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
50
Overview - OCR

1. Goal
2. System Architecture
3. Progress

51
Goal

• Reading text from signs labels

52
Approach

• Camera OCR software
• (OCR Optical Character Recognition)

53
High-level Architecture
Image from Camera
remote server
phone
Text
54
Architecture
55
System Evaluation
56
Assumptions

• The sight impaired user has a notion of where the
  camera should be pointed (ex. in line at
  restaurant, in front of billboard)
• Cameras picture is relatively straight on
• No perspective correction transformations applied
• Text is of a reasonable size

57
Trinetra OCR(end)

• Tudor Dumitras
• Kunal Kapur
• Pablo Quinones
• Matt Lee

58
Trinetra Transportation Group

• Puneet Aggarwal
• Chris DeLeon
• Aaron Paulos
• Joe Tadduni
• Andrew Williams
• Jovan Williams

59
Local Database
Bluetooth
OCR
GPRS/EDGE
USB
Inventory
Picture of text from phone
OCR Server (ABBYY Software)
Cell Phone Functions

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Built-in camera

Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS GPRS
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
Laptop (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
60
System Architecture
61
Software Architecture
62
Shuttle Hardware

• Computing Platform
• Fujitsu Lifebook P-series Laptop
• GPS Coordinate Receiver
• Garmin GPS 18
• Communication
• PCMCIA GPRS Cellular Modem
• Capacity Sensing
• Phidget 8/8/8 USB sensor interface board
• 2 Paired IR Transmitter / Receiver Circuits

63
Shuttle Software
64
End-user Hardware
End-user Hardware

• Supports high speed connectivity via GPRS
• Supports CLASS-4 EDGE
• standard for high-speed data transmission on
  GSM-based networks/phones
• Supports TALK software
• Web interface was developed using XHTML for
  Mobile 1.0

65
User Interaction
66
Assumptions Made

• End-user interaction
• No concurrency problems
• SMS gets to user in timely fashion
• No connection interruptions
• No invalid inputs
• Server
• Server is reliable and scalable
• MySQL database is thread safe
• Shuttle
• Perfect GPRS connection
• Application is fault free
• One shuttle per route is active at any given time

67
Demo

• Real points captured by GPS unit
• Fed into architecture on a 1/5 time scale
• Total of 21 stops and 60 way-points
• Represents the CMU Shuttle AB route
• Consists of 6 distinct subgroups
• User phone
• MySQL database for central coordination and data
  storage
• Tomcat Server for user presentation
• Java applet for SMS user notification
• C application that runs shuttle input handler,
  driver notifications, and capacity calculations
• C bus application that collects GPS points and
  routes to server

68
Trinetra Transportation Group(end)

• Puneet Aggarwal
• Chris DeLeon
• Aaron Paulos
• Joe Tadduni
• Andrew Williams
• Jovan Williams

69

• Design Guidelines

70
Safety Considerations

• Safety and reliability
• affected every decision

71
Design Guidelines

• Aim for effortless independence
• Leverage existing technologies
• Complement basic tools, dont complicate
• Reliability for real-life usability
• Unobtrusive, low-profile use
• Make system configurable

72

• Demo

73
Questions
74
Functional Capabilities

• Object Information
• Product prices, personal notes eg. serving size,
  expiration date, medication side effects
• Object Identification
• Cereal boxes on shelf, bottles in fridge
• Querying shuttle status
• Knows how long before next will arrive
• Able to notify driver that user will be waiting
• Awareness of location during shuttle ride
• Bus speaker calls out stops
• Awareness of menu options
• In a restaurant, user can compare menu options
  to posted specials
• Bus signs and information

Shopping
Transportation
Optical Character Recognition