CS 88188 Computational Models of Human Behavior

1
CS 88/188Computational Models of Human Behavior

• http//www.cs.dartmouth.edu/tanzeem/teaching/CS18
  8-Fall08/
• Instructor Tanzeem Choudhury
• tanzeem.choudhury_at_dartmouth.edu

2
Acknowledgement

• These slides include material from different
  talks and papers to illustrates some of the ideas
  we will cover in the class they include some of
  my work but also work done by my collaborators
  and students
• Matthai Philipose, Jim Rehg, James Kitts, Jeff
  Bilmes, Lin Liao, Maryam Mahdaviani, Jianxin Wu,
  Danny Wyatt, Jonathan Lester, Dieter Fox, Henry
  Kautz, Gaetano Borriello

3
Introductions

• Tell us a bit about
• yourself
• your research interests
• what you want to get out of this class

4
Why do we want to build systems that can
interpret human behavior?

• Necessary for building technology that is
• Assistive
• Context aware
• Natural user interface

5
Eldercare -Activities of Daily Living (ADLs)

• Classes of day-to-day activities that
• Indicate cognitive well-being
• Encompass most activities an elder may perform

6
What is grandma doing?

• Professional caregivers must
• assess elders competence
• fill out rating form
• Tedious, error-prone, expensive
• Fill ADL forms automatically?

7
Autism

• Kids with autism do not learn normal
  conversational protocols, e.g. turn-taking,
  intonation, personal space, etc. They often do
  not perceive objects or understand what the
  objects signify. They often have high I.Q. and
  work very well with technology and interested in
  modifying their behavior.

8
How is this done currently?

• behaviorally-based interventions as opposed to
  pharmacological have been found to be by far the
  most effective means of intervention
• Treatment for autistic children involve (a)
  high-density reinforcement to prime social
  responding (c) social skills training (d)
  adult-mediated prompting and reinforcement (e)
  self-monitoring
• Continuous or incremental monitoring and
  intervention impossible child-specific
  interventions for children with autism may, in
  isolation, have limited potential. The more
  intense the intervention, the greater the gain.

9
Future of mobile computing

• What are you doing?
• What do you like?
• Who do you hang out with?

10
What makes people tick?

• Why do some people become more influential than
  others?
• Why are some people more well-connected than
  others?

11
How to define activities?
Mom has an active and healthy lifestyle
activities
Mom eats regularly
Mom exercises regularly

Eat breakfast
Eat lunch
Stretch
Biking

actions
Get on Bike
Pedal

operations
12
Building people-aware systems

• Sensing gather information about actions and
  context
• Modeling develop features and algorithms that
  are useful for recognizing activities
• Applications make activity information
  actionable

rfid e00700017fab778 rfid e00700017fab783
13
What can activity recognition system do for you?
monitor
Logging What did they do? Rating How
well? Anomaly detection What was
wrong? Trending How have they
changed? Notification Call me when they need
me Prompting Walk them through
it Actuating (Help) do it.
act
14
Challenge Getting useful information is tough
privacy
ergonomics
scalability
15
New ideas that let us address the sensor challenge
Application
Tracker

• RFID tags and readers allow a simple, robust
  sensor stack for object-person interactions

Object ID
RFID
Object Recognition F (lighting
conditions, Object being detected, Kinematic
signatures, etc.)
Other Sensor Signals
16
Radio Frequency Identification (RFID)
Tiny, 40-cent battery-free tags
ambient readers
Tags return 96-bit ID when queried by readers
ltID e3f000e13431, desc bread basket,
manufacturer , gt
17
iBracelet Proximity Use

• 13.56MHz reader, radio, power supply, antenna
• 12 inch range, 12-150 hr lifetime
• Objects tagged on grasping surfaces

18
WISP Wireless ID Sensing Platform

• Passive RFIDSensors
• small stickers
• battery-free
• room-range

motion sensor wisp
19
Mobile sensing platform (MSP)

• Multi-modal instead of multiple locations
• Complementary modalities
• audio, acceleration, temperature, light,
  humidity, pressure compass
• Wearable
• always with user, controlled by user, compact

20
Embedded processing and communication

• Desirable for interactive applications
• Big advantages in terms of privacy

MSP sensing, data processing, and
communication
21
A Five Minute Data Trace
22
Acceptable privacy solution is important

• Nobody wants to record everything they say
• need to protect peoples privacy and privacy of
  anyone they come into contact with
• Process audio on the fly so that it is impossible
  to reconstruct the words later

23
Privacy-sensitive processing of speech
paralinguistic information

• Record and store information about how
• Throw away information about what

24
Privacy-sensitive modeling can infer meaningful
attributes of interaction

• speaking time interruptions
• speaking rate, changes in loudness pitch
• differences in speaking rate
• intonation, turn-taking
• turn-taking, loudness, interruptions

• status, dominance roles
• emotional state
• bi-polar disorder, depression
• autism, sociability
• conversation types

25
Privacy sensitive features

• For detecting speech
• relative spectral entropy
• of auto-correlation peaks
• non-initial max auto-correlation peak
• For segmenting speakers
• pair-wise energies
• For detecting un-micd speakers
• entropy of energy across mics

frequency
26
A few features commonly used
27
Challenges in modeling human behavior

• Reduce the amount of human effort required in
• feature engineering
• labeled data for training
• Develop models that are
• rich yet tractable
• parameters the are interpretable
• smooth
• adapts over time

28
Automatically selecting useful features
All the features you can think of
Features useful for classification
29
Why not use all the features?

• High dimensional data means
• learning a huge number of parameters
• training becomes extremely expensive
• and resulting models become brittle

30
Example classification
31
Want to combine the benefits

• Discriminative feature selection capability of
  boosting
• Ability to encode dependencies over time

32
Temporal information results in smoother
classification
33
Reduce dependency on human provided labels
tissue, razor, toothpaste, shaving cream, razor,
television, couch, remote, couch, dishwasher,
cabinet, sink, cabinet, silverware, dishwasher,
bathroom, light, toothpaste, toothbrush, cabinet,
cleaner, cleaner, cleaner, cabinet, toothbrush,
light, bathroom, highchair diaper, clothes,
towel, highchair, towel, diaper
34
Can we automatically mine common sense?
unlabeled data (objects used)
text
generic models
mine
label
labeled data

• Techniques for mining generic activity models
  from the web

learn
Joint work with Matthai Philipose and Danny Wyatt
35
Mining Models from the Web
how to pages
coffee water spoon milk sugar
make coffee
extract objects
find how to pages
compute usage probability
p(use spoon make coffee)
assemble model
Joint work with Matthai Philipose and Danny Wyatt
36
Bootstrapping a Customized Model
unlabeled data (objects used)
text
generic models
mine
label
labeled data
learn
Use mined model to label unlabeled traces Learn
new model from those labeled traces
Joint work with Matthai Philipose and Danny Wyatt
37
Dealing with incompleteness

• Preparing pasta

pot
spoon
kitchen range
spaghetti
stove
macaroni
pan
fork
38
WordNet ontology generation
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
stove
pot
spaghetti
39
WordNet ontology expansion
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
stove
pot
spaghetti
40
WordNet ontology expansion
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
knife
fork
stove
microwave
hotplate
pot
pan
poacher
spaghetti
linguine
macaroni
41
Acquire labels from complementary channels

• Leverage synergy between different sensor
  modalities
• RFID provides sparse and noisy labels
• Use these labels to train object model from video
• Provide basic information about objects likely to
  be used in a given activity
• making tea involves teabag, teacup, hot water
• Fuse RFID, vision, and prior knowledge to improve
  recognition

42
Infer object usage from sparse RFID measurements

• RFID measurements Rt
• Object in use Ot

43
Incorporate commonsense information

• Prefer objects that are usually used in activity
  At

44
Learn object models without manual labeling

• Use the marginal probabilities of object usage to
  train object histograms without any human
  provided labels

45
Put everything together
46
Objects and activities modeled

• Water jug
• Kettle
• Teabag
• Cup
• Spoon
• Milk
• Honey
• Cereal
• Bowl
• Coffee
• Creamer
• Sugar
• Cheese
• Bread
• Knife
• Toaster

• Plate
• Butter
• Peanut butter
• Jelly
• Lunch bag
• Plant
• Plant care
• Watering can
• Pillbox
• Salad tosser
• Salad dressing
• Meat
• Microwave
• Popcorn
• Juice
• Cloth
• phone

• Boiling water
• Making tea
• Preparing cereal
• Making coffee
• Making cheese sandwich
• Making buttered toast
• Making peanut butter sandwich
• Packed lunch
• Tending plants
• Taking medicine
• Making TV dinner
• Making salad
• Making popcorn
• Drinking juice
• Wiping counter
• Making a phone call

• Objects Activities

47
Future opportunities

• Need to
• further reduce the amount of human engineering
  and effort required
• develop systems that adapt over time
• take advantage of the community of users

48
Goal of the seminar

• Learn about the various machine learning
  approaches developed to make computers more aware
  of people, their activities, and their
  surrounding context.
• Discuss the various research challenges in data
  collection, representation and tractability of
  models, and evaluation.
• Brain-storm ideas on how future research can go
  about tackling some of these challenges.

49
Written Critique

• Email a one-page written critique for each paper
  - due at 6pm the day before the class
• put CS 188 assignment in your subject line
• The critique should
• identify the technical challenges the researchers
  are trying to solve
• describe the new ideas
• discuss the strength and limitations of the
  proposed approach
• suggest potential improvements

50
Presentation

• Email me your top 3 choices by end of today
• Dont just summarize the sections
• Be critical yet constructive
• Slides matters spend time on the structure and
  the overall presentation
• Dont be afraid to be controversial
• Have some questions ready to spur discussion

51
Project

• You are encouraged to choose a topic (with my
  approval) that matches your interests
• Encouraged to do projects in groups of two but
  individual projects are also acceptable
• Group projects need to have roughly twice the
  contribution of an individual project
• You will be graded on the novelty and depth of
  your idea as well as the implementation

52
Two different flavors of projects (1/2)

• Implement some of the main ideas from a paper you
  read in class but
• explore a new application scenario or
• create a new and compelling demo

53
Two different flavors of projects (2/2)

• Propose a new algorithm (or an extension to an
  existing algorithm)
• you won't be penalized if your idea does not give
  you the result you expect
• you should make a convincing argument for why you
  think your idea will be successful in your
  proposal
• you need to devote enough time exploring why the
  idea did not work

54
Project proposal due 10/14

• Submit a 1-2 page proposal
• Proposal should answer the following questions
• what problem are you trying to solve?
• why is it important or useful?
• what are the hard problems and how will solve
  them?
• how will you evaluate/measure success?
• For a multi-person project, please include how
  you will divide the tasks amongst yourselves
• Include a timeline

55
Mid-term project update 11/4

• Each student or group is required to do a
  mid-term project presentation
• this is your chance to get feedback from everyone
• Submit a written update on the progress
• highlight any unexpected problems or results
• are you on track given the original timeline
• I will give you individual feedback

56
Final project presentation due 12/2 and 12/3

• Each student or group is required to do a 15
  minute final project presentation
• And submit a 10-15 page written report or a
  formatted for submission to a conference
• if you are interested in doing a conference paper
  thats great but come talk to me first

57
Grading

• Class Attendance Participation 10
• Written Critiques 20
• Presentations 20
• Project 50
• Late submissions not accepted
• if there is an emergency contact me before an
  assignment is due
• each student has one grace assignment (written
  critiques only) - inform me in advance

58
Honor Code

• I encourage you to discuss a paper with your
  classmate but do not collaborate on the actual
  writing of the critiques
• You are permitted to use available machine
  learning toolboxes for your project but you need
  to explicitly reference the resources you use
• You project needs to include significant coding
  beyond the use of available toolboxes
• You must reference all other sources of help and
  collaboration

59
Last but not least

• There will be a best project award!
• You will get to cast your vote
• The best project will be a weighted combination
  of my vote and your cumulative vote ?