Decision Making AI

1
Decision Making AI

• John See
• 20 Dec 2010

2
Decision Making AI

• Ability of a game character to decide what to do
• Decision Making in Millingtons Model

3
Decision Making AI

• Decision Trees
• Finite State Machines (FSM)
• Rule-based Systems
• Fuzzy Logic Neural Networks
• Blackboard Architecture

4
Decision Trees

• Fast, easy to understand
• Simplest technique to implement, but extensions
  to the basic algorithm can be sophisticated
• Typically used to control characters, animation
  or other in-game decision making
• Can be learned, and learning is relatively fast
  (compared to fuzzy logic/NN)

5
Decision Trees Problem Statement

• Given a set of knowledge, we need to generate a
  corresponding action from a set of actions
• Map input and output typically, a same action
  is used for many different sets of input
• Need a method to easily group lots of inputs
  together under one particular action, allowing
  the input values that are significant to control
  the output

6
Decision Trees Problem Statement

• Example Grouping a set of inputs under an action

Enemy is visible Enemy is now lt 10m away
Attack
Enemy is visible Enemy is still far (gt 10m), but
not at flank
Attack
Enemy is visible Enemy is still far (gt 10m), at
flank
Move
Enemy is not visible, but audible
Creep
7
Decision Trees Algorithm Overview

• Made up of connected decision points
• Tree has starting decision, its root
• For each decision, starting from the root, one of
  a set of ongoing options is chosen.
• Choice is made based on characters knowledge
  (internal/external) ? Fast! No prior
  representation!
• Continues along the tree, making choices at each
  decision node until no more decisions to consider
• At each leaf of the tree, an action is attached
• Action is carried out immediately

8
Decision Trees

• Check a single value and dont contain any
  Boolean logic (AND, OR)
• Representative set
• Boolean Value is true
• Enumeration Matches one of the given set of
  values
• Numeric value Value is within given range
• 3D Vector Vector has a length within given
  range
• Examples?

9
Decision Trees Combining Decisions

• AND two decisions place in series in the tree
• OR two decisions also use decisions in series,
  but the two actions are swapped over from the AND
  

10
Decision Complexity

• Number of decisions that need to be considered is
  usually much smaller than number of decisions in
  the tree.
• Imagine using IF-ELSE statements to test each
  decision?
• Method of building DTs Start with simple tree,
  as AI is tested in game, additional decisions can
  be added to trap special cases or add new
  behaviors

11
Decision Making - Branching

• So far, we have considered only binary trees
  decisions choose between 2 options.
• It is possible to build DT with any number of
  options, or different decisions with different
  number of branches

12
Decision Making - Branching

• Deep DTs may result in a same alert being checked
  numerous times before a decision is found
• Flat DTs are more efficient, requires less
  decision checking

13
Decision Making - Branching

• Still common to find DTs using only binary
  decisions
• Why?
• Underlying code for multiple branches simplifies
  down to a series of binary tests (IF-ELSE
  statements)
• Binary DTs are easier to optimize. Some learning
  algorithms that work with DTs require them to be
  binary

14
Decision Making Performance

• Takes no memory, performance is linear with
  number of nodes visited
• Assume each decision takes constant amount of
  time, and tree is balanced, performance O(log2
  n), where n is number of decision nodes in tree
• This DOES NOT consider the execution time of the
  different checks required in the DT, which can
  vary a lot!

15
Decision Making Balancing the Tree

• DTs can run the fastest when a tree is balanced
• A balanced tree keeps the height of its branches
  approximately equal (within 1 to be considered
  balanced). In our context, it will have about the
  same number of decision making levels on each
  branch

16
Decision Making Balancing the Tree

• 8 behaviors, 7 decisions
• 1st tree extremely unbalanced, 2nd tree
  balanced
• To get to H, 1st tree needs 8 decisions, 2nd tree
  needs 3 only

17
Decision Making Balancing the Tree

• If all behaviors are equally likely, what is the
  average number of decisions needed for both trees?

18
Decision Making Balancing the Tree

• If we were likely to end up at decision A
  majority of time, which is more efficient?
• How do we treat decisions that are time-consuming
  to run? Lets say A is the most time-consuming
  decision

19
Decision Making Merging Patterns

• DTs can be extended to allow multiple branches to
  merge into a new decision efficient, but some
  care is needed!
• A decision/action can be reached in more than one
  way
• Avoid causing loops, can never find an action
  leaf

20
Random Decision Trees

• To provide some unpredictability and variation to
  making decisions in DTs
• Simplest way Generate a random number and choose
  a branch based on its value
• DTs are normally intended to run frequently,
  reacting to the game state, random decisions can
  be a problem
• What is a potential problem with the following
  DT?

21
Random Decision Trees

• Possible considerations
• Allow random decision to keep track of what it
  did last time.
• When a decision is considered, a choice is made
  at random, and that choice is stored. Next time
  the decision is considered, no more randomness,
  previous choice is maintained.
• If something in the world changes, a different
  decision was arrived, the stored choice should be
  removed.
• Timing Out Allow the AI to time out after a
  set time, and a random choice is to be made
  again. Gives variety and realism.

22
State Machines

• Often, characters in a game act in one of a
  limited set of ways
• Carry on doing the same thing until some event or
  influence makes them change
• Can use decision trees, but it is easier to model
  this behavior using state machines (or finite
  state machines, FSM)
• State machines take into consideration
• the world around them
• their internal state

23
State Machines Basics

• Each AI character occupies one state at each
  instance
• Actions or behaviors are associated with each
  state
• So long as the character remains in that state,
  it will continue carrying out the same
  actions/behavior
• States are connected by transitions
• Each transitions leads from one state to another,
  the target state, and each has a set of
  associated conditions
• Changing states when the game determines that
  conditions of a transition are met, the
  conditions trigger and a new state is fired

24
State Machines Simple Example

• State machine to model a soldier 3 states
• Each state has its own transitions
• The solid circle (with a transition w/o trigger
  condition) points to the initial state that will
  be entered when the state machine is first run

25
State Machines vs. Decision Trees

• Now, name some obvious differences in making
  decisions using decision trees and state
  machines?

26
Finite State Machines (FSM)

• In game AI, a state machine with this kind of
  structure (as seen earlier) is usually called a
  finite state machine (FSM)
• An FSM has a finite number of states and
  transitions
• It has finite internal memory to store its states
  and transitions

27
FSM Generic Implementation

• Use a generic state interface that keeps track of
  a set of possible states and records the current
  state it is in
• With each state, a series of transitions are
  maintained. Each transition is also a generic
  interface with conditions
• At each iteration (game loop), an update function
  is called to check if any of the transitions from
  the current state is triggered.
• If a transition is triggered, then the transition
  will be fired
• The separation of triggering and firing of
  transitions allows the transitions to have their
  own actions

28
FSM Generic Implementation

• Refer to textbook or other references for a more
  in-depth code-level implementation of FSMs

29
FSM Complexity

• State machines only requires memory to hold a
  triggered transition and the current state
• O(1) in memory and O(m) in time, where m is the
  (average) number of transitions per state
• The algorithm calls other supporting functions to
  perform action and etc. These probably account
  for most of the time spent in the algorithm.
• Hard-coded FSM inflexible, does not allow level
  designers the control over building the FSM logic

30
Hard-coded FSM

• Hard-coded FSM
• Consists of an enumerated value, indicating which
  state is currently occupied, and a function that
  checks if a transition is followed
• States are HARD-CODED, and limited to what was
  HARD-CODED
• Pros Easy and quick implementation, useful for
  small FSMs
• Cons
• Inflexible, does not allow level designers the
  control over building the FSM logic
• Difficult to maintain (alter) Large FSMs, messy
  code
• Every character needs to be coded its own AI
  behaviors

31
Hierarchical State Machines

• One state machine is a powerful tool, but will
  still face difficulty expressing some behaviors
• Also if you wish to model somewhat different
  behaviors from more than one state machines for a
  single AI character
• Example Modeling alarm behaviors with
  hierarchical s/m (using a basic cleaning robot
  state machine)

32
Hierarchical State Machines

• If the robot needs to get power if it runs out of
  power resume its original duties after
  recharging, these transition behaviors must be
  added to ALL existing states to ensure robot acts
  correctly

33
Hierarchical State Machines

• This is not exactly very efficient. Imagine if
  you had to add many more concurrent behaviors
  into your primary state machine?

34
Hierarchical State Machines

• A hierarchical state machine for the cleaning
  robot
• Nested states could be in more than one state
  at a time
• States are arranged in a hierarchy ? next state
  machine down is only considered when the higher
  level state machine is not responding to its alarm

35
Hierarchical State Machines

• H - history state node
• When the composite state (lower hierarchy) is
  first entered, the H node indicates which
  sub-state should be entered
• If composite state already entered, then previous
  sub-state is restored using the H node

36
Hierarchical State Machines

• Hierarchical state machine with cross hierarchy
  transition
• Most hierarchical s/m support transitions between
  levels of the hierarchy
• Lets say we want the robot to go back to refuel
  when it does not find any more trash to collect

37
Hierarchical State Machines

• Refer to textbook for more details on its
  implementation
• Performance
• O(n) in memory (n is number of layers in
  hierarchy)
• O(nt) in time, where t is number is number of
  transitions per state

38
DT SM

• Combining decision trees and state machines
• One approach Replace transitions from a state
  with a decision tree
• Leaves of DT (rather than straightforward
  conditions/actions) are now transitions to other
  states

39
DT SM

• To implement state machine without decision tree
  transitions
• We may need to model complex conditions that
  require more checking per transition
• May be time-consuming as need to check all the
  time

40
Fuzzy Logic

• Founded by Lotfi Zadeh (1965)
• the essence of fuzzy logic is that everything is
  a matter of degree
• Imprecision in data
• and uncertainty in solving problems
• Fuzzy logic vs. Boolean logic
• 50-80 less rules than traditional rule-based
  systems, to accomplish identical tasks
• Examples Air-conditioner thermostat or washing
  machine

41
Fuzzy Logic in Games

• Example of uses
• To control movement of bots/NPCs (to smooth out
  movements based on imprecise target areas)
• To assess threats posed by players (to make
  further strategic decisions)
• To classify player and NPCs in terms of some
  useful game information (such as combat or
  defensive prowess)

42
Crisp data Fuzzy data

• Crisp data (real numbers, value)
• Fuzzy data (a predicate or description, with
  degree value)
• Fuzzy logic gives a predicate a degree value.
  Instead of belonging to a set of being excluded
  (1 or 0, Boolean logic), everything can partially
  belong to a set, and some things more belong than
  others

43
Fuzzy sets

• Fuzzy sets the numeric value is called the
  degree of membership (these values are NOT
  probability values!)
• For each set, a degree of membership of 1 given
  to something completely in the set. Degree
  membership of 0 given to something completely
  outside the fuzzy set
• Typical to use integers in implementation instead
  of floating-point values (between 0 and 1), for
  fast computation in game
• Note Anything can be a member of multiple sets
  at the same time

44
Fuzzy Control / Inference Process

• 3 basic steps in a fuzzy control or fuzzy
  inference process

45
Step 1 - Fuzzification

• Mapping process converts crisp data (real
  numbers) to fuzzy data (degree of membership)
• E.g. Given a persons weight, find the degree to
  which a person is underweight, overweight or at
  ideal weight

46
Membership Functions

• Membership functions map input variables to a
  degree of membership, in a fuzzy set between 0
  and 1
• Any function can be used, and the shape usually
  is governed by desired accuracy, the nature of
  problem, or ease of implementation.
• Boolean logic m/f

47
Membership Functions

• Grade m/f
• Reverse grade m/f
• Triangular m/f
• Trapezoid m/f

48
Membership Functions

• Earlier example of using a set of membership
  functions to represent a persons weight

49
Step 2 Fuzzy rule base

• Once all inputs are expressed in fuzzy set
  membership, combine them using logical fuzzy
  rules to determine degree to which each rule is
  true
• E.g.
• Given a persons weight and activity level as
  input, define rules to make a health decision
• If overweight AND NOT active then frequent
  exercise
• If overweight AND active then moderate diet
• But having a fuzzy output such as frequency
  exercise is not enough need to quantify the
  amount of exercise (e.g. 3 hours per week)

50
Fuzzy rules

• Usually uses IF-THEN style rules
• If A then B
• A ? antecedent / premise
• B ? consequent / conclusion
• To apply usual logical operators to fuzzy input,
  we need the following fuzzy axioms
• A OR B MAX(A, B)
• A AND B MIN(A, B)
• NOT A 1 A

51
Fuzzy rules

• Earlier example on weight (and now, including
  height)
• Note that these fuzzy axioms (AND, OR, NOT) are
  not the only definition of the logical operators.
  There are other definitions that can be used

overweight AND tall MIN(0.7, 0.3)
0.3 overweight OR tall MAX(0.7, 0.3) 0.7 NOT
overweight 1 0.7 0.3 NOT tall 1 0.3
0.7 NOT (overweight AND tall) 1 MIN(0.7, 0.3)
0.7
52
Complete Rule Base

• With the above m/f for each input variable,
  common requirement is to construct a complete set
  of all possible combination of inputs. In this
  case, we need 18 rules (2x3x3)

53
Rule evaluation (Creature example)

• We have an AI fuzzy decision making system, which
  needs to evaluate whether a creature should
  attack the player. Input variables range,
  health, opponent ranking

54
Rule evaluation (Creature example)

• Rule base
• If (in melee range AND uninjured) AND NOT hard
  then attack
• If (NOT in melee range) AND uninjured then do
  nothing
• If (NOT out of range AND NOT uninjured) AND (NOT
  wimp) then flee
• Given specific degrees for the input variables,
  we might get outputs that are like
• Attack degree 0.2
• Do nothing degree 0.4
• Flee degree 0.7

55
Rule evaluation (Creature example)

• So what do we do with those fuzzy membership
  output values??
• Missing link We also need to represent the
  output variable as a fuzzy membership set!

56
Step 3 Defuzzification

• Defuzzification process Fuzzy output ? Crisp
  output
• From previous step, each rule in rule base
  results in a degree of membership in some output
  fuzzy set
• With the numerical output we got earlier (0.2 for
  attack, 0.4 for do nothing, 0.7 for flee), we
  shall construct a composite output membership
  function

57
Step 3 Defuzzification

• Not possible to be exact/accurate, but there are
  methods that solve the problem as near as
  possible
• Using Highest Membership Function
• Choose fuzzy set which has highest degree of
  membership and choose the output value that
  represents each set.
• 4 common points min, max, average of min/max,
  bisector
• Very simple to implement but coarse
  defuzzification

58
Step 3 Defuzzification

• Blending Based on Membership
• Blend each characteristic point based on its
  corresponding degree of membership
• E.g. Character with 0.2 attack, 0.4 do nothing,
  0.7 flee will produce crisp output given by (0.2
  attack direction) (0.4 do nothing
  direction) (0.7 flee direction)
• Make sure that the eventual result is normalized
  (otherwise result may be over-the-bounds or
  unrealistic)
• Common normalization technique Divide total
  blended sum by the sum of fuzzy output values
• Minimum values blended (Smallest of Maximum, SoM)
• Maximum values blended (Largest of Maximum, LoM)
• Average values blended (Mean of Maximum, MoM)

59
Step 3 Defuzzification

• Center of Gravity
• Also known as Centroid of Area method ? Takes
  into account all membership values, rather than
  specific ones (largest, smallest, average, etc.)
• First, each m/f is cropped at the membership
  value of its set
• Center of mass is found by integrating each in
  turn. This point is chosen as output crisp value
• Unlike bisector of area method, we cant compute
  this offline since we do not know in advance the
  fuzzy membership values, and how the m/f will be
  cropped

60
Misc Dealing with Complex Rule Base

• We may have multiple rules in our rule base that
  will results in the same output membership fuzzy
  set.
• E.g.
• Corner-entry AND going-slow THEN accelerate
• Corner-exit AND going-fast THEN accelerate
• Corner-exit AND going-slow THEN accelerate
• How do we deal with such situations? Which output
  membership value for accelerate to choose?

61
Some Good Examples

• 2 Examples Control Example Threat Assessment
  Example from AI for Game Developers ref book