Title: High-level robot behavior control using POMDPs Joelle Pineau and Sebastian Thrun Carnegie Mellon University
1High-level robot behavior control using
POMDPsJoelle Pineau and Sebastian
ThrunCarnegie Mellon University
2Introducing Pearl
- Pearl is a prototype nursing robot, providing
assistance to both nurses and elderly people.
3Profile of our aging population
- 450,000 more nurses needed by 2008
- campaign to recruit and retain nurses and
other health care providers
4Our vision of robotic-assisted health-care
Management support of ADLs
Providing information (TV, weather)
Reminding to eat, drink, take meds
Monitoring Rx adherence safety
Linking the caregiver to resources
Calling for help in emergencies
Providing physical assistance
Supporting inter-personal communication
Moving things around
Enabling use of remote health services
5The Nursebot project in its early days
6We need a high-level controller that can
- prioritize goals from specialized modules
- trade-off goals with widely different
costs/rewards - trade-off between information-gathering and
goal- satisfaction - switch between tasks and share sensory
information - handle uncertainty
7High-level robot behavior control usingPartially
Observable Markov Decision Processes
BELIEF STATE
OBSERVATIONS
STATE
USER WORLD ROBOT
ACTIONS
8What are POMDPs?
POMDP is n-tuple S, A, ?, b, T, O, R
POMDP task 1 State tracking After an
action, what is the state of the world? POMDP
task 2 Computing a policy Which action should
the controller apply next?
rt-1
rt
st-1
st
...
...
World state
ot
ot-1
at-1
??
Control layer
bt-1
??
...
...
Robot belief
Not so hard.
Very hard!
9Our approach Hierarchical POMDPs
- Key Idea Exploit hierarchical structure in the
problem domain to break a problem into many
related POMDPs.
subtask
Act
abstract action
InvestigateHealth
Move
Navigate
AskWhere
CheckPulse
CheckMeds
North
South
East
West
primitive action
10Planning with Hierarchical POMDPs
- Given POMDP model M S, A, ?, b, T, O, R and
hierarchy H - For each subtask h ? H
- 1) Set components
- Ah ? children nodes
- Sh ? S
- ?h ? ?
- bh, Th, Oh, Rh
- 2) Minimize model
- Sh ? zh(s0), , zh(sn)
- ?h ? yh(o0), , yh(op)
- 3) Solve subtask h
- ?h ? bh, Th, Oh, Rh
AMove AskWhere,Navigate SMove
X,Y,Destination ?Move o0,,op
ANav N,S,E,W SNav X,Y ?Nav o0,,om
Move
Navigate
AskWhere
South
East
West
North
11Execution with Hierarchical POMDPs
- Step 1 - Update belief
- Step 2 - Traversing hierarchy top-down, for each
subtask - 1) Get local belief.
- 2) Consult local policy.
- 3) If a is leaf node, terminate.
- Else, go to that subtask.
12Experimental Setup
- Task Robot provides reminders and guidance to
elderly user. - Action hierarchy
13Sample Scenario
14Still unconvinced about the importance of
uncertainty?
15On the question of numerical vs logical
representations...
- We havent tried using logical representations to
control the robot - But our experience tells us that
- Uncertainty is crucial when dealing with people.
- Probabilistic techniques are necessary to reason
about uncertainty. - Real belief tracking and planning really
matters!