CS 88188 Computational Models of Human Behavior - PowerPoint PPT Presentation

Loading...

PPT – CS 88188 Computational Models of Human Behavior PowerPoint presentation | free to download - id: 1c4506-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

CS 88188 Computational Models of Human Behavior

Description:

These s include material from different talks and papers ... Cooking Utensil. Food. Kitchen Appliance. pasta. pot. stove. Entity. Artifact. Cutlery. solid ... – PowerPoint PPT presentation

Number of Views:23
Avg rating:3.0/5.0
Slides: 60
Provided by: tanzeemc
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: CS 88188 Computational Models of Human Behavior


1
CS 88/188Computational Models of Human Behavior
  • http//www.cs.dartmouth.edu/tanzeem/teaching/CS18
    8-Fall08/
  • Instructor Tanzeem Choudhury
  • tanzeem.choudhury_at_dartmouth.edu

2
Acknowledgement
  • These slides include material from different
    talks and papers to illustrates some of the ideas
    we will cover in the class they include some of
    my work but also work done by my collaborators
    and students
  • Matthai Philipose, Jim Rehg, James Kitts, Jeff
    Bilmes, Lin Liao, Maryam Mahdaviani, Jianxin Wu,
    Danny Wyatt, Jonathan Lester, Dieter Fox, Henry
    Kautz, Gaetano Borriello

3
Introductions
  • Tell us a bit about
  • yourself
  • your research interests
  • what you want to get out of this class

4
Why do we want to build systems that can
interpret human behavior?
  • Necessary for building technology that is
  • Assistive
  • Context aware
  • Natural user interface

5
Eldercare -Activities of Daily Living (ADLs)
  • Classes of day-to-day activities that
  • Indicate cognitive well-being
  • Encompass most activities an elder may perform

6
What is grandma doing?
  • Professional caregivers must
  • assess elders competence
  • fill out rating form
  • Tedious, error-prone, expensive
  • Fill ADL forms automatically?

7
Autism
  • Kids with autism do not learn normal
    conversational protocols, e.g. turn-taking,
    intonation, personal space, etc. They often do
    not perceive objects or understand what the
    objects signify. They often have high I.Q. and
    work very well with technology and interested in
    modifying their behavior.

8
How is this done currently?
  • behaviorally-based interventions as opposed to
    pharmacological have been found to be by far the
    most effective means of intervention
  • Treatment for autistic children involve (a)
    high-density reinforcement to prime social
    responding (c) social skills training (d)
    adult-mediated prompting and reinforcement (e)
    self-monitoring
  • Continuous or incremental monitoring and
    intervention impossible child-specific
    interventions for children with autism may, in
    isolation, have limited potential. The more
    intense the intervention, the greater the gain.

9
Future of mobile computing
  • What are you doing?
  • What do you like?
  • Who do you hang out with?

10
What makes people tick?
  • Why do some people become more influential than
    others?
  • Why are some people more well-connected than
    others?

11
How to define activities?
Mom has an active and healthy lifestyle
activities
Mom eats regularly
Mom exercises regularly

Eat breakfast
Eat lunch
Stretch
Biking

actions
Get on Bike
Pedal

operations
12
Building people-aware systems
  • Sensing gather information about actions and
    context
  • Modeling develop features and algorithms that
    are useful for recognizing activities
  • Applications make activity information
    actionable

rfid e00700017fab778 rfid e00700017fab783
13
What can activity recognition system do for you?
monitor
Logging What did they do? Rating How
well? Anomaly detection What was
wrong? Trending How have they
changed? Notification Call me when they need
me Prompting Walk them through
it Actuating (Help) do it.
act
14
Challenge Getting useful information is tough
privacy
ergonomics
scalability
15
New ideas that let us address the sensor challenge
Application
Tracker
  • RFID tags and readers allow a simple, robust
    sensor stack for object-person interactions

Object ID
RFID
Object Recognition F (lighting
conditions, Object being detected, Kinematic
signatures, etc.)
Other Sensor Signals
16
Radio Frequency Identification (RFID)
Tiny, 40-cent battery-free tags
ambient readers
Tags return 96-bit ID when queried by readers
ltID e3f000e13431, desc bread basket,
manufacturer , gt
17
iBracelet Proximity Use
  • 13.56MHz reader, radio, power supply, antenna
  • 12 inch range, 12-150 hr lifetime
  • Objects tagged on grasping surfaces

18
WISP Wireless ID Sensing Platform
  • Passive RFIDSensors
  • small stickers
  • battery-free
  • room-range

motion sensor wisp
19
Mobile sensing platform (MSP)
  • Multi-modal instead of multiple locations
  • Complementary modalities
  • audio, acceleration, temperature, light,
    humidity, pressure compass
  • Wearable
  • always with user, controlled by user, compact

20
Embedded processing and communication
  • Desirable for interactive applications
  • Big advantages in terms of privacy

MSP sensing, data processing, and
communication
21
A Five Minute Data Trace
22
Acceptable privacy solution is important
  • Nobody wants to record everything they say
  • need to protect peoples privacy and privacy of
    anyone they come into contact with
  • Process audio on the fly so that it is impossible
    to reconstruct the words later

23
Privacy-sensitive processing of speech
paralinguistic information
  • Record and store information about how
  • Throw away information about what

24
Privacy-sensitive modeling can infer meaningful
attributes of interaction
  • speaking time interruptions
  • speaking rate, changes in loudness pitch
  • differences in speaking rate
  • intonation, turn-taking
  • turn-taking, loudness, interruptions
  • status, dominance roles
  • emotional state
  • bi-polar disorder, depression
  • autism, sociability
  • conversation types

25
Privacy sensitive features
  • For detecting speech
  • relative spectral entropy
  • of auto-correlation peaks
  • non-initial max auto-correlation peak
  • For segmenting speakers
  • pair-wise energies
  • For detecting un-micd speakers
  • entropy of energy across mics

frequency
26
A few features commonly used
27
Challenges in modeling human behavior
  • Reduce the amount of human effort required in
  • feature engineering
  • labeled data for training
  • Develop models that are
  • rich yet tractable
  • parameters the are interpretable
  • smooth
  • adapts over time

28
Automatically selecting useful features
All the features you can think of
Features useful for classification
29
Why not use all the features?
  • High dimensional data means
  • learning a huge number of parameters
  • training becomes extremely expensive
  • and resulting models become brittle

30
Example classification
31
Want to combine the benefits
  • Discriminative feature selection capability of
    boosting
  • Ability to encode dependencies over time


32
Temporal information results in smoother
classification
33
Reduce dependency on human provided labels
tissue, razor, toothpaste, shaving cream, razor,
television, couch, remote, couch, dishwasher,
cabinet, sink, cabinet, silverware, dishwasher,
bathroom, light, toothpaste, toothbrush, cabinet,
cleaner, cleaner, cleaner, cabinet, toothbrush,
light, bathroom, highchair diaper, clothes,
towel, highchair, towel, diaper
34
Can we automatically mine common sense?
unlabeled data (objects used)
text
generic models
mine
label
labeled data
  • Techniques for mining generic activity models
    from the web

learn
Joint work with Matthai Philipose and Danny Wyatt
35
Mining Models from the Web
how to pages
coffee water spoon milk sugar
make coffee
extract objects
find how to pages
compute usage probability
p(use spoon make coffee)
assemble model
Joint work with Matthai Philipose and Danny Wyatt
36
Bootstrapping a Customized Model
unlabeled data (objects used)
text
generic models
mine
label
labeled data
learn
Use mined model to label unlabeled traces Learn
new model from those labeled traces
Joint work with Matthai Philipose and Danny Wyatt
37
Dealing with incompleteness
  • Preparing pasta

pot
spoon
kitchen range
spaghetti
stove
macaroni
pan
fork
38
WordNet ontology generation
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
stove
pot
spaghetti
39
WordNet ontology expansion
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
stove
pot
spaghetti
40
WordNet ontology expansion
Entity
Substance
Object
Artifact
solid



Cutlery
Food
Cooking Utensil
Kitchen Appliance
pasta
spoon
knife
fork
stove
microwave
hotplate
pot
pan
poacher
spaghetti
linguine
macaroni
41
Acquire labels from complementary channels
  • Leverage synergy between different sensor
    modalities
  • RFID provides sparse and noisy labels
  • Use these labels to train object model from video
  • Provide basic information about objects likely to
    be used in a given activity
  • making tea involves teabag, teacup, hot water
  • Fuse RFID, vision, and prior knowledge to improve
    recognition

42
Infer object usage from sparse RFID measurements
  • RFID measurements Rt
  • Object in use Ot

43
Incorporate commonsense information
  • Prefer objects that are usually used in activity
    At

44
Learn object models without manual labeling
  • Use the marginal probabilities of object usage to
    train object histograms without any human
    provided labels

45
Put everything together
46
Objects and activities modeled
  • Water jug
  • Kettle
  • Teabag
  • Cup
  • Spoon
  • Milk
  • Honey
  • Cereal
  • Bowl
  • Coffee
  • Creamer
  • Sugar
  • Cheese
  • Bread
  • Knife
  • Toaster
  • Plate
  • Butter
  • Peanut butter
  • Jelly
  • Lunch bag
  • Plant
  • Plant care
  • Watering can
  • Pillbox
  • Salad tosser
  • Salad dressing
  • Meat
  • Microwave
  • Popcorn
  • Juice
  • Cloth
  • phone
  • Boiling water
  • Making tea
  • Preparing cereal
  • Making coffee
  • Making cheese sandwich
  • Making buttered toast
  • Making peanut butter sandwich
  • Packed lunch
  • Tending plants
  • Taking medicine
  • Making TV dinner
  • Making salad
  • Making popcorn
  • Drinking juice
  • Wiping counter
  • Making a phone call
  • Objects Activities

47
Future opportunities
  • Need to
  • further reduce the amount of human engineering
    and effort required
  • develop systems that adapt over time
  • take advantage of the community of users

48
Goal of the seminar
  • Learn about the various machine learning
    approaches developed to make computers more aware
    of people, their activities, and their
    surrounding context.
  • Discuss the various research challenges in data
    collection, representation and tractability of
    models, and evaluation.
  • Brain-storm ideas on how future research can go
    about tackling some of these challenges.

49
Written Critique
  • Email a one-page written critique for each paper
    - due at 6pm the day before the class
  • put CS 188 assignment in your subject line
  • The critique should
  • identify the technical challenges the researchers
    are trying to solve
  • describe the new ideas
  • discuss the strength and limitations of the
    proposed approach
  • suggest potential improvements

50
Presentation
  • Email me your top 3 choices by end of today
  • Dont just summarize the sections
  • Be critical yet constructive
  • Slides matters spend time on the structure and
    the overall presentation
  • Dont be afraid to be controversial
  • Have some questions ready to spur discussion

51
Project
  • You are encouraged to choose a topic (with my
    approval) that matches your interests
  • Encouraged to do projects in groups of two but
    individual projects are also acceptable
  • Group projects need to have roughly twice the
    contribution of an individual project
  • You will be graded on the novelty and depth of
    your idea as well as the implementation

52
Two different flavors of projects (1/2)
  • Implement some of the main ideas from a paper you
    read in class but
  • explore a new application scenario or
  • create a new and compelling demo

53
Two different flavors of projects (2/2)
  • Propose a new algorithm (or an extension to an
    existing algorithm)
  • you won't be penalized if your idea does not give
    you the result you expect
  • you should make a convincing argument for why you
    think your idea will be successful in your
    proposal
  • you need to devote enough time exploring why the
    idea did not work

54
Project proposal due 10/14
  • Submit a 1-2 page proposal
  • Proposal should answer the following questions
  • what problem are you trying to solve?
  • why is it important or useful?
  • what are the hard problems and how will solve
    them?
  • how will you evaluate/measure success?
  • For a multi-person project, please include how
    you will divide the tasks amongst yourselves
  • Include a timeline

55
Mid-term project update 11/4
  • Each student or group is required to do a
    mid-term project presentation
  • this is your chance to get feedback from everyone
  • Submit a written update on the progress
  • highlight any unexpected problems or results
  • are you on track given the original timeline
  • I will give you individual feedback

56
Final project presentation due 12/2 and 12/3
  • Each student or group is required to do a 15
    minute final project presentation
  • And submit a 10-15 page written report or a
    formatted for submission to a conference
  • if you are interested in doing a conference paper
    thats great but come talk to me first

57
Grading
  • Class Attendance Participation 10
  • Written Critiques 20
  • Presentations 20
  • Project 50
  • Late submissions not accepted
  • if there is an emergency contact me before an
    assignment is due
  • each student has one grace assignment (written
    critiques only) - inform me in advance

58
Honor Code
  • I encourage you to discuss a paper with your
    classmate but do not collaborate on the actual
    writing of the critiques
  • You are permitted to use available machine
    learning toolboxes for your project but you need
    to explicitly reference the resources you use
  • You project needs to include significant coding
    beyond the use of available toolboxes
  • You must reference all other sources of help and
    collaboration

59
Last but not least
  • There will be a best project award!
  • You will get to cast your vote
  • The best project will be a weighted combination
    of my vote and your cumulative vote ?
About PowerShow.com