An Extended Behavior Network for a Game Agent: Investigating Action Selection Quality and Agent Perf

1
An Extended Behavior Network for a Game Agent
Investigating Action Selection Quality and Agent
Performance in Unreal Tournament
MICAI05 - Mexican International Conference on
Artificial Intelligence

• Hugo Pinto
• Luis Otavio Alvares
• hsspinto, alvares_at_inf.ufrgs.br

Universidade Federal do Rio Grande do Sul
Instituto de Informática
2
Summary
MICAI05 - Mexican International Conference on
Artificial Intelligence

• Motivation
• Unreal Tournament (UT)
• Extended Behavior Networks
• Action Selection Quality
• Agent Performance
• Conclusion

3
Why computer games?(Engineer)
MICAI05 - Mexican International Conference on
Artificial Intelligence

• The computer game industry has beaten the movies
  industry in the USA, regarding revenue, for 2
  consecutive years .
• The governments of Australia, Brazil and South
  Korea have special programs supporting the
  research and development of computer games.

4
Why computer games?(Scientist)

• Many problems posed by computer games resemble
  the problems faced in robotics and other
  artificial intelligence fields a decade ago.
• It is time to not only design new solutions, but
  to see if old solutions apply to these new but
  similar problems and adapt them to these new
  domains.

5
Unreal Tournament

• 3D, first-person.
• Real time.
• Continuous(aproximatelly)
• Large action repertory
• Overall goal Kill enemies and stay alive.

6
(No Transcript)
7
Action Selection Problem

• In the design of a game robot one of the key
  concerns is how it selects its actions so as to
  exhibit a goal-oriented behavior.
• Search approaches are complicated due to the
  large state-space.
• Traditional planning is made more difficult as
  the outcomes of actions are uncertain and the
  environemnt is dynamic.

8
Behavior Networks

• Behavior Networks were proposed as action
  selection mechanisms to select good enough
  actions in complex and dynamic environments
• They combine properties of traditional planners
  and connectionist systems.

9
Good Enough?

• Maes defined a good enough action selection
  policy as one that has the following
  characteristics favors actions that contribute
  to the agents goals (specially several goals at
  once), favors actions that contribute to ongoing
  plans, exploit opportunities, is fast, is robust
  and avoids conflicts among goals.

10
Extended Behavior Networks(1)

• Behavior Networks have been constantly evolving
  since their first appearance(1989), with five
  major extensions to the original model between
  1990 and 2003.
• Extended Behavior Networks represent the
  state-of-the-art in Behavior Network models.

11
Extended Behavior Networks (2)

• Defined by a static structure and an action
  selection algorithm.
• Structure
• Goals
• Behaviors
• Resources
• Control Parameters

12
Behavior Network Structure (1)
EnemyHurt 0.8
Low Ammo Not LowAmmo 0.6
EnemyInSight ShootEnemy EnemyHurt 0.6,
LowAmmo 0.1 Hands 2, Head 1
Head f(s)1
Hands f(s) 2
Legs f(s)2
13
Behavior Network Structure (2)
EnemyHurt 0.8
Low Ammo Not LowAmmo 0.6
g
?
(Inhibition influence)
(Activation influence)

b
(Inertia)
EnemyInSight ShootEnemy EnemyHurt 0.6,
LowAmmo 0.1 Hands 2, Head 1
,
q
(Activation Threshold)
Dq
(Decay of Activation Threshold)
Hands f(s) 2
Head f(s)1
Legs f(s)2
14
Action Selection Algorithm

• 1) For each module, calculate its activation.
• 2) For each module, calculate its executability.
• 3) For each module, calculate its execution-value
  .
• 4) Each module, for each resource it has not
  bound yet, tries to bind the resource.
• 5) If a module has bound all its resources it
  resets the values of the resources it has bound
  to the value of the global threshold and
  executes.
• 6) Upon finishing execution a module unbinds its
  resources.
• 7) If a resource has not been bound by any module
  it decays its threshold ?res by ??.
• 8) Return to step 1.

15
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (1)
16
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (2)
17
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (3)
18
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (4)
19
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (5)
20
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action chaining (6)
21
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
6
3
2
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
5
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
1
2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
1
4
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Action Chaining (7)
22
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
3
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
2
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
1
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Reactivity and Persistence
23
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Conflict Resolution
24
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
ltNo preconditionsgt Stand EnemyInSight
0.1 Legs 2
Not EnemyInSight Explore EnemyInSight
0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
Preference for actions that satisfy more than
one goal
25
Performance

• Two Experiments
• 1) In the first we tested our agent (EBN_Bot)
  against a totally different agent built by
  another group (CMU_JBot)
• 2) The second experiment used a robot that had
  identical sensors and behaviors but a different
  action selection algorithm.

26
Table 1 EBN Bot X CMU JBot
27
First Experiment

• The EBN agent scored much higher.
• The low number of killings is due to an absence
  of a chasing behavior in both agents.
• Is the better performance due to our sensors and
  behaviors or due to the action selection
  mechanism used? The next experiment sheds light
  on this issue.

28
Second Experiment (1)

• In this experiment we compared the EBN agent to
  an agent that had exactly the same sensors and
  behaviors but used a different action selection
  strategy
• At each time step we disregard activation
  spreading for action selection and take into
  account only the executability of each module.

29
Second Experiment(2)

• Now that we do not have activation we are faced
  frequently with situations in which two modules
  have the same execution-value.
• We need to hard code some priority rules or
  insert additional conditions to decide which one
  to launch when appropriate.
• We have opted for the first approach in most
  cases, to differ as little as possible from the
  original behavior network.

30
EnemyHurt 0.9
Low Ammo Not LowAmmo 0.8
Not HaveHighHealth HaveHighHealth 0.8
IAmBeingShot Not IAmBeingShot 0.9
IAmBeingShot Dodge Not IAmBeingShot 0.3 Legs
2
MedKitReachable GetReachableMedKit HaveHighHeal
th 0.6 Legs 2
EnemyNear FinalizeWithHammer EnemyHurt 0.8
Hands 2, Head 1
EnemyInSight ShootEnemy EnemyHurt 0.5,
LowAmmo 0.2 Hands 2, Head 1
Not EnemyInSight StopShooting Not LowAmmo
0.1 Hands 2
EnemyInSight GoToEnemy EnemyNear
0.6, IAmBeingShot 0.2 Legs 2
MedKitInSight GoToMedKitInSight MedKitReachable
0.8 Legs 2
Not EnemyInSight StandLookout EnemyInSight
0.2 MediKitInSight 0.1 Legs 2, Head 1
Not EnemyInSight, Not IAmBeingShot Explore Enem
yInSight 0.3 MedKitInSight 0.2 Legs 2, Head 1
KnowMedKitLocation GoToKnownMedKit MedKitInSigh
t 0.8 Legs 2
31
(No Transcript)
32
Discussion

• One interesting point is that our robot had 100
  more killings but just a little over 10 more
  hits. This is due to the quality of its action
  chains.
• Another point that catches attention is that the
  mean difference in total score was much smaller
  in this experiment

33
Conclusion

• The superior performance of the robot using
  extended behavior networks in both experiments
  makes the case for extended behavior networks as
  a good and competitive solution to action
  selection for game agents.
• The assessment of the quality of the actions
  selected contributes to validate extended
  behavior networks as an action selection
  mechanism for complex agents with many goals
  situated in complex, dynamic and continuous
  environments.

34
Future Work

• Incorporate continuous learning of action
  outcomes.
• Investigate ways of automatically setting the
  global parameters.
• Extending the network to allow other game modes.

35
Game Over

• Thank you!
• More information on current and future works at
  www.hugopinto.net.
• You may contact us at
• hugo_at_hugopinto.net
• alvares_at_inf.ufrgs.br