Senior Capstone Project Gaze Tracking System what you see is finally what you get

1
Senior Capstone ProjectGaze Tracking Systemwhat
you see is finally what you get

• Team IRALAR
• Breanna Heidenburg -- Michael Lenisa --
  Daniel Wentzel
• Advisor Dr. Malinowski

2
Presentation Outline

• The Project
• Why is it important
• The Goals
• System breakdown
• Image recognition
• Point transformation
• User Interface
• The Results

3
Brief overview

• What is our project?

4
Brief Overview

• Track a users eye and use the information to
  control a computer cursor.

Wherever you look on the display is where the
cursor goes
5
Brief Overview

• Enhances Human Computer Interaction
• Speed of use
• Hands-free use

6
System Development

• 3 Part System
• Image Processing Application
• Calibration and Mapping system
• GUI designed for gaze-based interaction
• Systems developed concurrently and independently
• Separate Applications at run-time

7
System Development

• Hardware and Image Processing Application

8
System Development Software
Software
Image Processing Application
Hardware (Input)
Hardware (output)
User Interface Application
Image Processing Cursor Control Thread
UI Thread
OS channel
Shared Process Data
UDP channel
UDP Server Thread
9
System Development Hardware
Software
Hardware (Input)
Hardware (output)
10
System Development Hardware (input)

• Hardware
• Camera
• QuickCam Pro for Notebooks
• Visible Spectrum Camera
• Polarizer
• Tiffen 25mm polarizing filter
• Removes glare from eye reflections
• Lighting
• Diffused LED
• Slightly distracting to the user, but necessary
  to provide light for the camera

11
System Development Hardware
Software
Hardware (Input)
Hardware (output)
12
System Development Hardware (output)

• LitEye LE-500
• High resolution (SVGA)
• Color Display
• Translucent or opaque operation
• Stationary relative to users eye

13
System Development Software
Software
Software
Hardware (Input)
Hardware (output)
Image Processing Application
14
System Development Software Image Processing

• Real time pupil tracking system
• Developed in C using OpenCV image processing
  libraries
• Traditional image processing and blob tracking
• Capabilities
• Locate and determine center of pupil in image
• Low light and high reflection environments
• All eye colors
• Data logging and static test modes
• Packaged into self contained Windows installer
  for easy deployment onto any computer

15
System Development Software Image Processing
Query Frame From Camera
Reject False Positives
Calculate Blob Center
Red Channel
Smooth
Contrast Stretch
Extract Blobs
Final Recognition
16
System Development Software Image Processing

• Summary
• The Good
• Dynamically adapts to changing lighting
  conditions and eye types
• Maintains performance in low-light and specularly
  noisy conditions
• The Bad
• Still relies on Logitech camera drivers
• Extreme reflections still cause problems

17
System Development Image Processing

• Examples of performance in poor conditions
• Low Light Difficult False Positive

Note Image brightness and contrast artificially
enhanced for human visibility
18
System Development

• Calibration and point mapping

19
System Development User Interface
Software
Image Processing Application
Hardware (Input)
Hardware (output)
Image Processing Cursor Control Thread
20
System Development Eye Mapping

• System for mapping the location of the center of
  the pupil to a pixel on a computer screen
• Must Calibrate for each User
• Geometry
• The eye is not flat but a screen is
• User Customization
• All eyes are different
• Everyone wears the HMD differently
• User Training
• Calibration system also acts as a quick tutorial

21
System Development Point Transformation

• 3 dimensional best fit plane
• Currently using a 4th degree best fit
• Xpix A1 XeyeB1 YeyeC1
• Ypix A2 XeyeB2 YeyeC2
• Calibration sub-system determines these
  coefficients

22
System Development Calibration

• How do we solve the problem?
• Multiple Variable Linear Regression Least
  Squares
• Y B0 B1x1 Bkxk
• Uses matrix algebra to obtain a coefficient
  matrix
• B (XX)-1XY

23
ResultsCalibration
Error Mesh

• Cursor position error
• actual vs. determined position
• Horizontal and vertical error in screen pixels

24
ResultsCalibration

• Accuracy varies with position and skill of user
• Corners of screen most difficult to calibrate
• Focusing on a rapidly changing location requires
  skill
• Limitations
• 1 Accuracy of human vision system
• Eye Saccades
• Original error goal
• 2 of screen dimension on both axes
• Achieved error
• 1.18 Horizontal
• 1.46 Vertical

25
System Development Clicking

• How do we click?
• Monitor eye movements
• Identify pauses
• Dwell time - When eye position is focused on a
  single area for a period of time
• Currently set at 5 frames (200 mS)
• Generally, it takes 230 mS for a hand to click a
  mouse.

26
ResultsTiming

• Improved interaction speed
• Trackpad vs. Mouse vs. Gaze Tracking
• 53 Increase over Trackpad
• 12 Increase over Traditional Mouse

27
System Development

• Custom GUI interface and Communications

28
System Development User Interface
Software
Image Processing Application
Hardware (Input)
Hardware (output)
User Interface Application
Image Processing Cursor Control Thread
UI Thread
29
System Development User Interface

• Custom GUI for Gaze Tracking Applications
• Why?
• Gaze tracking accuracy limited by inherent
  properties of the human vision system
• Traditional GUI too small and intrusive for use
  with transparent HMD
• Demonstrate applications of gaze tracking

30
System Development User Interface

• Modern GUI Design
• WPF using XAML layout
• Windows Presentation Foundation
• eXtensible Application Markup Language
• XAML is similar to HTML
• Uses tags and code-behind in a similar style to
  javascript
• GUI coded in C

31
System Development User Interface

• Multiple pages within the interface
• Screens for functionality testing
• even games
• Ability to minimize interface

32
System Development User Interface
Main Menu Screen

• Large text and buttons
• Wide spacing between options
• Simple layout
• Placement of features in high-accuracy areas

33
System Development Software
Software
Image Processing Application
Hardware (Input)
Hardware (output)
User Interface Application
Image Processing Cursor Control Thread
UI Thread
OS channel
34
System Development Communication

• Why
• Allows processes to communicate
• Allows relay of time sensitive information
• 2 Communication Channels
• OS Channel
• Omni-directional (Image Processing to User
  Interface)
• UDP Channel
• Multi-directional
• Separate Thread in Image Processing Application

35
System Development Communication
OS channel
Shared Process Data
UDP channel
UDP Server Thread
Creates Multi-threading issue
36
System Development Communication

• Multi-threading
• Public variable usage
• Solution Critical Section
• Raises thread priority (thread is
  uninterruptable)

Receive request for data (over UDP)
Raise Thread priority
Lower Thread Priority
Reply to request
Read Variable
37
System Development Software
Software
Image Processing Application
Hardware (Input)
Hardware (output)
User Interface Application
Image Processing Cursor Control Thread
UI Thread
OS channel
Shared Process Data
UDP channel
UDP Server Thread
38
System Development Software
39
Results

• Goals met and future projects

40
ResultsCore Goals

• Image Processing based eye tracking system
• Correlates eye position to location on screen
• Adapts to wide variety of eye types
• Ability to function as a hands-free input device
• Cursor control via eye-tracking system
• Display visual output to user
• Visually controlled demo programs

41
ResultsAdditional Goals

• Augmented reality system
• Transparent Heads-up display over user vision
• Portable system for everyday use
• Forward looking camera for correlation of user
  real-world vision to eye position
• Real world applications
• Network integration
• Sound output

42
ResultsFinal Package

• Installation Screen

• Windows compatible software package
• Standard Windows Installer
• Contains image processing application and GUI

43
ResultsFuture Projects

• Pupil Tracking with Neural Network
• Implementation of camera driver with DirectShow
• Implement Head tracking for gaze-tracking without
  head-mounted display
• Augmented reality
• Front facing camera
• Object/face recognition
• Implement real-world applications
• Any one of these is a senior project in itself!

44
ResultsSchedule
45
Thanks

• Who helped us out?

46
Thanks

• Dr. Malinowski and the EE faculty
• Mr. Mattus Mr. Schmidt
• Our test subjects

47
Questions