Search

1
Search

• Can we apply the recently discussed search
  methods too all problems?
  
• Why or why not?
• How does the search change when there are
  multiple actors involved?

2
Game Playing

• A second player (or more) is an adversary
• Examples chess, checkers, go, tic-tac-toe,
  connect four, backgammon, bridge, poker
  
• Adversarial search.
• Players take turns or alternate play
• Unpredictable opponent.
• Solution specifies a move for every possible
  opponent move.
  
• Turn time limits
• How might they affect the search process?

3
Types of Games
4
Example Game Nim

• Deterministic, Opposite Turns
• Two players
• One pile of tokens in the middle of the table.
• At each move, the player divides a pile into two
  non-empty piles of different sizes.
  
• Player who cannot move looses the game.

5
Example Game Nim
min
7
Assume a pile of seven tokens.
Max attempts to maximize the advantage and win.
max
5-2
6-1
4-3
min
3-3-1
3-2-2
4-2-1
5-1-1
Min always tries to move to a state that is the
worst for Max.
2-2-2-1
max
3-2-1-1
4-1-1-1
2-2-1-1-1
min
3-1-1-1-1
max
6
Perfect Decision, Two person games

• Two players
• MAX and MIN
• MAX moves first alternate turns thereafter.
• Formal definition of game
• Initial State
• Successor Function
• Terminal Test
• Utility Function
• No one player has full control, must develop a
  strategy.

7
Minimax Algorithm Basics
MAX
3
MIN
0
3
2
MAX
7
0
9
3
6
2
MIN
6
5
7
2
1
3
9
0
4
5
2
Process of Backing up minimax decision
Assumption Both players are knowledgeable and
play the best possible move
8
Minmax Algorithm

• Generate game tree to all terminal states.
• Apply utility function to each terminal state.
• Propagate utility of terminal states up one
  level.
  
• MAXs turn MAX tries to maximize utility value.
  
  
• MINs turn MIN minimizes utility value.
• Continue backing-up values toward root, one layer
  at a time.
  
• At root node, MAX chooses the value with highest
  utility.
  

9
Minimax Search

• What is the time complexity of the minimax
  algorithm?
  
• Minimax decision maximizes utility for a player
  assuming that the opponent will play perfectly
  and minimize its utility.
  
• m looking ahead m moves in game tree
• Search can be done in a depth first manner
• i.e. traverse down a path till terminal node
  reached, then back up value using minimax
  decision function.

10
Minimax Applied to Nim
min
7
1
5-2
6-1
4-3
max
1
1
1
min
3-3-1
1
0
3-2-2
4-2-1
1
5-1-1
0
max
2-2-2-1
0
3-2-1-1
0
1
4-1-1-1
1
2-2-1-1-1
min
0
3-1-1-1-1
What do we observe from tree?
max
2-1-1-1-1-1
0
11
Partial Search

• What were our concerns with the other search
  algorithms we have discussed?
  
• How do those concerns apply to games?

• Full game tree may be too large to generate.

12
Example Partial Search for Tic-Tac-Toe

• Generate tree to intermediate depth.
• Calculate utility/evaluation function for leaf
  nodes.
  
• Back up values to the root node so MAX player can
  make a decision.
  

13
Example Partial Search for Tic-Tac-Toe
Position p win for Max, u(p) ? loss for MAX,
u(p) ? otherwise, u(p) ( of complete row
s, cols, diags still open for MAX) ( of
complete rows, cols, and diags still open for MIN)
14
Tic-Tac-Toe Move 2
15
MAX Player 3rd move
16
Question
If we apply an evaluation function to
non-terminal nodes of a game tree, why generate a
partial tree at all? Why not just compute an ev
aluation function to all states generated by
applying valid operators (moves) to the current
state, and selecting the one with the highest
utility value?
Assumption An evaluation function applied to
successor nodes produces more reliable (accurate)
values. In other words, these configurations co
ntain more information on how the game will
likely end.
Each level of game tree termed a ply.
17
Complex games

• What happens if Minimax is applied to large
  complex games?
  
• What happens to the search space?
• Example, chess
• decent amateur program ? 1000 moves /second
• 150 seconds /move (tournament play)
• Look at approx. 150,000 moves
• Chess branching factor of 35
• generating trees that are 3-4 ply,
• Resultant play pure amateur

18
Complex Games

• Can we make search process more intelligent, and
  explore more promising parts of tree in greater
  depth?

19
MAX Player 3rd move
Search starts depth first
Back up value immediately
Do not need to generate these nodes! Why?
MIN
The alpha-beta pruning procedure
Node A generated first
20
Alpha-Beta Pruning

• Definitions
• ? value lower bound on MAX node
• ? value upper bound on MIN node
• Observations
• ? value on MAX nodes never decrease
• ? values on MIN nodes never increase
• Application
• Search is discontinued below any MIN node with
  min-value v ? ? ? cut off
  
• Search is discontinued below any MAX node with
  max-value v ? ? ? cut off

21
Alpha-Beta Search Example
MAX
a
b
c
MIN
d
f
g
MAX
e
MIN
5
1
5
1
1
2
2
3
4
7
0
6
Node Alpha Beta a -8 8
Node Alpha Beta a 3 8
b -8 8
b -8 3
d -8 8
d 1 8
d 2 8
d 3 8
e -8 3
e 4 3 CUT-OFF
c 3 8
c 3 3 CUT-OFF
f 3 8
Completed
22
Key -8 negative infinity 8 positive
infinity The last value in a square is the final
value assigned to the specific variable, i.e. at
the end of the search Node As a 3.
23
Alpha-Beta Search Algorithm

• Computing ? and ?
• ? value of MAX node current largest final
  backed-up value of its successors.
  
• ? value of MIN node current smallest final
  backed-up value of its successors.

24
Alpha-Beta Search Algorithm
Start with AB(n -?, ?)
Alpha-Beta-Search(state) returns an action
v MAX-VALUE(state,-8, 8) return the action
in SUCCESSORS(state) with value v
MAX-VALUE(state, a, b) if TERMINAL-TEST(state)
then return UTILITY(state) v -8
for a, s in SUCCESSORS(state) do v
MAX(v, MIN-VALUE(s, a, b)) if v b then re
turn v a MAX(a, v) return v (a utility
value)
MIN-VALUE(state, a, b) if TERMINAL-TEST(state
) then return UTILITY(state) v
8 for a, s in SUCCESSORS(state) do v MI
N(v, MAX-VALUE(s, a, b)) if v rn v b MIN(b, v) return v (a utility va
lue)
25
Alpha-Beta Search Example 2
MAX
a
c
b
MIN
d
f
g
MAX
e
MIN
n
j
h
l
m
k
i
MAX
1
2
5
8
4
2
3
5
7
7
1
2
0
0
26
Alpha-Beta Search Example 3
MAX
a
b
c
MIN
d
f
g
MAX
e
MIN
o
h
j
l
n
k
m
i
MAX
1
4
2
-1
1
1
3
-4
5
-1
0
-5
1
0
0
-3
27
Alpha-Beta Search Performance

• Uses depth-first search procedure
• Search effectiveness depends on node ordering
• Worst case occurs if worst moves generated first
• Best case occurs if best moves generated first
  min value first for MIN nodes, and max value
  first for MAX nodes.
  
• (Is this always possible?)

28
Alpha-Beta Search Performance

• Search tree of depth d, b moves per node
• bd tip nodes Minimax search - O(bd)
• Best case
• look at only O(bd/2) tip nodes
• Effective branching factor reduces to so can
  double the depth of the search.
  
• Average case
• Number of nodes examined
• Search depth increase 4/3

Pruning does not affect final result
29
Games with chance

• Example, Games with dice
• Backgammon white can decide what his/her legal
  moves are, but cant determine blacks because
  that depends on what black rolls.
  
• Game tree must include chance nodes.
• How to pick best move?
• Cannot apply minimax directly.

30
Games with Chance

• For each possible move, compute an average or
  expected value.
  

Complexity O(bmnm) where n is the number of
distinct
values (e.g. dice rolls)
31
Expectimax example
32
Types of Games
33
Game AI Model
AI is given processor time
Execution Management
AI receives information
Group AI
Content Creation
Strategy
Scripting
World Interface
Character AI
AI has implications for Related technologies
Decision Making
Movement
Animation
Physics
AI is turned into on-screen action
Artificial Intelligence for Games Ian
Millington 2006
34
Game AI Schematic
AI Engine
Main Game Engine
Content Construction
Behavior database
AI specific tools
AI data is used to construct characters
Modeling package
World Interface
Level Loader
World interface Extracts relevant Game data
AI behavior manager
Level design tool
Game engine Calls AI each frame
Per-frame processing
AI receives data from the game and from its int
ernal information
Results of AI Are written back To game data
Package level data
Artificial Intelligence for Games Ian
Millington 2006
35
AI in Games

• Types of AI in games
• Hacks
• Heuristics
• Common Heuristics
• Most Constrained
• Do the most difficult thing first
• Try the most promising thing first
• Algorithms

Artificial Intelligence for Games Ian
Millington 2006
36
AI in Games

• Coordinating action
• Learning

• Path following
• Collision avoidance
• Finding paths
• World Representations
• Movement planning
• Decision making
• Tactical analysis
• Terrain analysis

Artificial Intelligence for Games Ian
Millington 2006
37
Board Games

• Best AI agents for Chess, Backgammon and Reversi
  employ dedicated hardware, algorithms and
  optimizations.
  
• Basic underlying algorithms are common across
  games.
  
• Most popular AI for board games is minimax
  family.
  
• Recently Memory-enhanced test driver (MTD)
  algorithms are gaining favor.

Artificial Intelligence for Games Ian
Millington 2006
38
Transposition Tables

• Transposition Tables (memory)
• Multiple board positions that are identical.
• Table contains board positions and results of a
  search from that position.
  

Artificial Intelligence for Games Ian
Millington 2006
39
Transposition Tables

• Hash game states
• An hashing algorithm can work but
• Zobrist Keys a set of fixed-length random bit
  patterns stored for each possible state of each
  possible location on the board.
  
• Chess 64 x 2 x 6 768 entries
• For each non-empty square, the Zobrist key is
  located and XORed with a running hash total.
  

Artificial Intelligence for Games Ian
Millington 2006
40
Transposition Tables

• What to store
• Minimax Store value and the depth used to
  calculate the value.
  
• Alpha-Beta store the value, if the value is
  accurate or fail-soft (because of pruning).
  
• Alpha pruning fail-low value
• Beta pruning fail-high value

Artificial Intelligence for Games Ian
Millington 2006
41
Transposition Tables

• Issues
• Path Dependency
• Some games have scores dependent upon the
  sequence of moves.
  
• Repetitions will be identically scored, so
  algorithm may disregard a winning move.
  
• Fix by using a Zorbist key for number of
  repeats.

Artificial Intelligence for Games Ian
Millington 2006
42
Transposition Tables

• Issues
• Instability
• The stored values fluctuate during the same
  search since values may be over written.
  
• Not guaranteed to get the same value on each
  lookup.
  
• In rare cases, could have an oscillating value
  situation.

Artificial Intelligence for Games Ian
Millington 2006
43
Memory-Enhanced Test (MT) Algorithm

• Require an efficient transposition table that
  acts as memory.
  
• Implemented as a zero-width AB negamax with a
  transition table.
  
• Negamax swaps and inverts the alpha and beta
  values and checks and prunes only against the
  Beta value.

Artificial Intelligence for Games Ian
Millington 2006
44
MTD Algorithm

• The driver routine repeatedly calls the MT
  Algorithm to zoom in on the correct minimax
  value and determine the next move.
  Memory-enhanced Test Drivers.

Artificial Intelligence for Games Ian
Millington 2006
45
MTD Algorithm

• Track the upper bound on the score value Gamma
• Let Gamma be the first guess of the score.
• Calculate another guess by calling the MT
  algorithm for the current board position, the
  maximum depth, zero for the current depth, and
  the gamma value.
• If the returned guess is not the same as gamma,
  then calculate another guess in order to confirm
  that the guess is correct. (usually limit the
  number of iterations)
• Return the guess as the score.

Artificial Intelligence for Games Ian
Millington 2006
46
Opening Books

• An opening book is a set of moves combined with
  information regarding how good the average
  outcome is when using the moves.
  
• Typically at the start of the game.
• Play books certain games have stages where
  plays can be employed.

47
State of the Art Games

• Traditional games
• Chess HITECH (Berliner, CMU) alpha-beta,
  horizon effect, 10 million positions per move,
  accurate evaluation functions defeated human
  grandmaster in 1987
• Deep Thought Hsu, Ananthraman, and others at
  CMU generated 720,000 chess positions per second
  won World Computer Chess Championship in 1989
  
• Hsu, Campbell and others, CMU to IBM more focus
  on fast computation parallel processing to
  solve complex problems evolution from Deep
  Thought to Deep Blue (1993)
• Deep Blue massively parallel 32 node RISC
  system distributed memory built by IBM. Each
  node has 8 VLSI chess processors therefore, 256
  processors working in tandem.
• First played Kasparov in 1996, won game 1, but
  eventually lost to Kasparov.

48
State of the Art Games

• May 1997 (Deeper Blue) rematch. Processor speed
  doubled, evaluation functions improved with help
  from a multitude of grandmasters (chief advisor
  Joel Benjamin, US chess champ)
• Deep(er) Blue was generating and evaluating up
  to 200 million chess positions per sec, i.e., 200
  billion moves every 3 minutes. Typical
  grandmaster computes only 500 moves in 3
  minutes.
• Result Deep Blue won 2, lost 1, drew 3.
• Ques Was Deep Blue really intelligent?

49
State of the Art Games

• Chess (continued) Quote from Feng-hsuing Hsu
• The previous version of Deep Blue, lost a match
  to Gary Kasparov in Philadelphia in 1996. But
  2/3rds of the way into that match, we had played
  to a tie with Kasparov. That old version of Deep
  Blue was already faster than the machine I had
  conjectured in 1985, and yet it was not enough.
  There was more to solving the Computer Chess
  Problem than just increasing the hardware speed.
  Since that match, we rebuilt Deep Blue from
  scratch, going through every match problem, and
  engaging grandmasters extensively in our
  preparations. Somehow, all that work cause
  Grandmaster Joel Benjamin, our chess advisor, to
  say, You know, sometimes Deep Blue plays chess.
  Joel could not distinguish with certainty Deep
  Blues moves from the moves of top grandmasters.

50
State of the Art Games

• Checkers Big breakthrough in AI Samuels in 1952
  at IBM played itself thousands of times and
  learned its own evaluation functions. Evaluation
  function was a weighted sum of a set of
  functions.
• Best checkers programs now as good as human
  champions
  
• Othello human players refuse to play computer
  games, who can play perfectly.
  
• Go branching factor of game 300, so straight
  search doesnt work. Have to use pattern
  knowledge bases.
  
• Does not match up to human champions

51
State of the Art Games

• Backgammon 2 dice ? 20 legal moves (can be 6000
  with 1-1 roll)
  
• depth 4 as depth increases, probability of
  reaching nodes decreases, so value of look ahead
  is diminished. (alpha-beta not a great idea).
  
• best backgammon game uses depth 2 search very
  good evaluation function.
  
• plays at world championship level.