Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP)

1
Graham KendallAutomated Scheduling, Optimisation
and Planning Research Group (ASAP)
MIU, July 2004
2
Contents

• Checkers Why was it considered beaten?
• Two approaches to Checkers
• Poker (if time)

3

• 1959. Arthur Samuel started to look at Checkers2
• The determination of weights through self-play
• 39 Features
• Included look-ahead via mini-max

2 Samuel A. Some studies in machine learning
using the game of checkers. IBM J. Res. Develop.
3 (1959), 210-229
4

• Samuelss program defeated Robert Nealy, although
  the victory is surrounded in controversy
• Was he state champion?
• Did he lose the game or did Samuel win?

5
Checkers Starting Position








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
6
Checkers Moves








1
2
3
4
5
6
8
7
Pieces can only move diagonally forward
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
7
Jumps are forced
Checkers Forced Jumps








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
8
Red (Samuels Program)








2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
Getting to the back row gives a King
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
9
Red (Samuels Program)








1
2
3
4
5
6
8
7
Forced Jump
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
10
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
11
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
12
Red (Samuels Program)








1
2
3
4
What Move?
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
13
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
14
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
15

• This was a very poor move.
• It allowed Samual to retain his Triangle of
  Oreo
• AND.. By moving his checker from 19 to 24 it
  guaranteed Samuel a King
• This questioned how strong a player Nealy really
  was

16
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
17

• This was a very poor move.
• It allowed Samual to retain his Triangle of
  Oreo
• AND.. By moving his checker from 19 to 24 it
  guaranteed Samuel a King
• This questioned how strong a player Nealy really
  was

18
Red (Samuels Program)








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
19
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
20
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
What Move (5, 13 or 16)?
27
29
30
31
32
White (Nealey)
21
Computers Game Playing A Potted History
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
22
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
23
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
24
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
25
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
26
Computers Game Playing A Potted History
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
27
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
28
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
This checker is lost
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
29
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
30
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
What Move (3, 6 or 19)?
29
30
31
32
White (Nealey)
31
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
This checker could run (to 10) but..
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
32
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
33
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
34
Red (Samuels Program) After Move 25








1
2
3
4
Forced Jump
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
35
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
36
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
37
Red (Samuels Program) After Move 25








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
White (Nealey)
38

• Two Mistakes by Nealy
• Allowing Samuel to get a King
• Playing a move that led to defeat when there was
  a draw available

39

• The next year a six match rematch was won by
  Nealy 5-1.
• Three years later (1966) the two world
  championship challengers (Walter Hellman and
  Derek Oldbury) played four games each against
  Samuels program. They won every game.

40

• Checkers
• Chinook
• Blondie 24 (aka Anaconda)

41
Types of Games

• Perfect
• Each Player has complete knowledge of the game
  state
• Usually only two players, who take alternate
  turns
• Examples include Chess, Checkers, Awari,
  Connect-Four, Go, Othello

42
Types of Games

• Imperfect
• Some of the game state is hidden
• Examples include Poker, Cribbage, Bridge

43
Types of Games

• Games with an element of chance
• The game moves have some stochastic element
• For example, Backgammon

44
Types of Games
Solved or Cracked Over Champion World Champion Grand-Master Amateur
Connect-Four Checkers (8x8) Chess Go (9x9) Go (19x19)
Qubic Othello Backgammon
Nine Mens Morris
Go_moku
Awari
6 Jaap van den Herik H., Uiterwijk and van
Rijswijck J. Games Solved Now and in the future.
Artificial Intelligence 134 (2002) 277-311
45
Case Study 1 Checkers

• Samuels work, perhaps, restricted the research
  into Checkers until 1989 when Jonathan Schaeffer
  began working on Chinook
• He had two aims
• To develop the worlds best checkers player
• To solve the game of checkers

46
Case Study 1 Checkers

• Chinook, at its heart, had an evaluation function
• Piece count (30 for a King)
• Runaway checker
• Dog Hole
• The weights were hand-tuned

47
Case Study 1 Checkers

• Opening game database from published work (with
  corrections they found)
• Initially 4000 openings, leading to an eventual
  40,000
• Cooks innovative lines of play that could
  surprise an opponent
• The aim was to take opponents into unknown
  territory

48
Case Study 1 Checkers

• Endgame database Started writing in May 1989
• The 8-piece endgame database finished on February
  20th 1994

49
Case Study 1 Checkers
1 120
2 6,972
3 261,224
4 7,092,774
5 148,688,232
6 2,503,611,964
7 34,779,531,480
8 406,309,208,481
50
Case Study 1 Checkers
9 4,048,627,642,976
10 34,778,882,769,216
11 259,669,578,902,016
12 1,695,618,078,654,976
13 9,726,900,031,328,256
14 49,134,911,067,979,776
15 218,511,510,918,189,056
16 852,888,183,557,922,816
51
Case Study 1 Checkers
17 2,905,162,728,973,680,640
18 8,568,043,414,939,516,928
19 21,661,954,506,100,113,408
20 46,352,957,062,510,379,008
21 82,459,728,874,435,248,128
22 118,435,747,136,817,856,512
23 129,406,908,049,181,900,800
24 90,072,726,844,888,186,880
TOTAL 500,995,484,682,338,672,639
52
Case Study 1 Checkers

• With a 4-piece database Chinook won the 1989
  Computer Olympiad
• In the 1990 US National Checkers Championship
  Chinook was using a 6-piece database.
• It came second, to Marion Tinsley, defeating Don
  Lafferty on the way who was regarded at the
  worlds second best player.

53
Case Study 1 Checkers

• Marion Tinsley
• Held the world championship from 1951 to 1994
• Before playing Chinook, Tinsley only lost 4
  competitive games (no matches)

54
Case Study 1 Checkers

• The winner of the US Championship has the right
  to play for the world championship. Finishing
  second (with Tinsley first) entitled Chinook to
  play for the world championship
• The American Checkers Federation (ACF) and the
  European Draughts Association (ADF) refused to
  let a machine compete for the title.

55
Case Study 1 Checkers

• In protest, Tinsley resigned
• The ACF and EDF, created a new world
  championship, man versus machine and named
  Tinsley as the world champion.
• At this time Tinsley was rated at 2812, Chinook
  was rated at 2706

56
Case Study 1 Checkers

• The match took place 17-29 Aug 1992.
• The 300,000 computer used in the tournament ran
  at about half the speed of a 1GHz PC
• The match finished 4-2 in favour of Tinsley (with
  34 draws)

57
Case Study 1 Checkers

• A 32 game rematch was held in 1994
• 8-piece end game
• Processors four times as fast (resulted in a
  factor of 2 speed up due to more complex
  evaluation function and the overhead of parallel
  processing)
• Opening book of 40,000 moves
• In preparation Chinook no losses in 94 games
  against Grandmasters

58
Case Study 1 Checkers

• Six games in (1-1, with 4 draws) Tinsley resigned
  for health reasons. His symptoms were later
  diagnosed as pancreatic cancer.
• Tinsley died on 3rd April 1995 (aged 68).
  Undoubtedly the best player ever to have lived
• Chinook was crowned the man versus machine
  champion. The first automated game player to have
  achieved this.
• A 20-match with Don Lafferty resulted in a draw
  (1-1, with 18 draws)

59
Case Study 1 Checkers








defeating the world who had held the title for
40 years
Opening Game Database (40,000) moves
Hand Crafted Evaluation Function (a/b search)
Schaeffer J. One Jump Ahead Challenging Human
Supremacy in checkers, Springer, 1997
Won the World (Man Versus Machine) Championship in
1994
Marion Tinsley lost his 5th, 6th and 7th games
to Chinook
End Game Database (8-pieces)
60
Case Study 2 Anaconda

• Project started in the summer of 1998, following
  a conversation between David Fogel and Kumar
  Chellapilla
• It was greatly influenced by the recent defeat of
  Kasparov by Deep Blue
• Chess was seen as too complex so draughts was
  chosen instead
• The aim is to evolve a player rather than build
  in knowledge

61
Case Study 2 Anaconda

• Reject inputting into a neural network what
  humans think might be important
• Reject inputting any direct knowledge into the
  program
• Reject trying to optimise the weights for an
  evaluation function

62
Case Study 2 Anaconda

• The Gedanken Experiment
• I offer to sit down and play a game with you. We
  sit across an 8x8 board and I tell you the legal
  moves
• We play five games, only then do I say You got 7
  points.I dont tell you if you won or lost
• We play another five games and I say You got 5
  points
• You only know higher is better

63
Case Study 2 Anaconda

• The Gedanken Experiment
• How long would it take you to become an expert at
  this game?
• We cannot conduct this experiment but we can get
  a computer to do it

64
Case Study 2 Anaconda

• Samuels Challenge Can we design a program that
  would invent its own features in a game of
  checkers and learn how to play, even up to the
  level of an expert?

65
Case Study 2 Anaconda

• Newells Challenge Could the program learn just
  by playing games against itself and receiving
  feedback, not after each game, but only after a
  series of games, even to the point where the
  program wouldnt even know which programs had
  been won or lost?
• Newell (and Minsky)7 believed that this was not
  possible, arguing that the way forward was to
  solve the credit assignment problem.

7 Minsky M. Steps Towards Artificial
Intelligence. Proceedings of the IRE, 1961, 8-30
66
Later changed to an explicit piece differential
Case Study 2 Anaconda
HL11
HL21
I1
Evaluation used for MiniMax
O
I32
HL210
HL140
weights1741
67
Case Study 2 Anaconda








1
2
3
4
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
68
Case Study 2 Anaconda
1
7
8
14
25
27
All other neurons have an value of zero
69
Case Study 2 Anaconda

• Algorithm
• Initialise 30 Networks
• Each network played 5 games as red against random
  opponents
• Games were played to completion or until 100
  moves had been made (a draw)
• 2 for a win, 0 for a draw, -1 for a loss
• 15 best performing networks were saved for the
  next generation and copies were mutated

70
Case Study 2 Anaconda

• Observations
• The points for a win, lose draw were set such
  that wins were encouraged. No experimentation
  with different values were tried
• Players could play a different number of games.
  This was, purposefully, not taken into account
• Mutation was carried out using an evolutionary
  strategy

71
Case Study 2 Anaconda

• After 10 Generations
• After 10 generations the best neural network was
  able to beat both its creators and a simple
  (undergraduate project) program which, by the
  authors admission was weak
• Note 400MHz PC

72
Case Study 2 Anaconda

• ACF Ratings

Grand (Senior) Master 2400 Class E 1000-1199
Master 2200-2399 Class F 800-999
Expert 2000-2199 Class G 600-799
Class A 1800-1999 Class H 400-599
Class B 1600-1799 Class I 200-399
Class C 1400-1599 Class J lt200
Class D 1200-1399
73
Case Study 2 Anaconda

• After 100 Generations
• Playing on zone.com
• Initial rating 1600
• Beat a player ranked at 1800 but lost against a
  player in the mid 1900s
• After 10 games their ranking had improved to
  1680. After 100 games it had improved to 1750
• Typically a 6-ply search but often 8-ply

74
Case Study 2 Anaconda

• Observations
• The highest rating it achieved was 1825
• The evolved King value was 1.4, which agrees with
  perceived wisdom that a king is worth about 1.5
  of a checker
• In 100 generations a neural network had been
  created that was competitive with humans
• It surpassed Samuels program
• The challenge set by Newell had been met

75
Case Study 2 Anaconda

• The Next Development
• Alpha-Beta Pruning introduced and evolved over
  250 generations
• Over a series of games, Obi_WanThe Jedi defeated
  a player rated at 2134 (48 out of 40,000
  registered) and also beat a player rated 2207
  (ranked 18)
• Final rating was 1902 (taking into account the
  different orderings of the games)

76
Case Study 2 Anaconda

• The Next Development
• Spatial nature of the board was introduced as at
  the moment it just saw the board as a vector of
  length 32

77
Case Study 2 Anaconda








1
2
3
4
36 3x3 Overlapping squares
5
6
8
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
78
Case Study 2 Anaconda








1
2
3
4
5
6
8
25 4x4 Overlapping squares
7
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
79
Case Study 2 Anaconda








1
2
3
4
5
6
8
7
16 5x5 Overlapping squares
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
80
Case Study 2 Anaconda








1
2
3
4
5
6
8
7
9 6x6 Overlapping squares
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
81
Case Study 2 Anaconda








1
2
3
4
5
6
8
7
4 7x7 Overlapping squares
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
82
Case Study 2 Anaconda








1
2
3
4
5
6
8
7
1 8x8 Overlapping squares
11
9
10
12
14
15
16
13
17
18
19
20
23
24
21
22
25
28
26
27
29
30
31
32
83
Case Study 2 Anaconda

• The Next Development
• 362516941 91 inputs
• 5,046 weights

84
Case Study 2 Anaconda
HL1 (91 nodes)
HL2 (40 nodes)
HL3 (10 nodes)
36 3x3
I1
25 4x4
16 5x5
O
9 6x6
I32
4 7x7
1 8x8
Sum of 32 Board Inputs
weights5046
85
Case Study 2 Anaconda

• 2 months and 230 generations later!!
• After 100 games the rating was 1929
• A 27 point increase over the previous network.
  Nice but not decisive
• Maybe it was due to there being three times more
  weights but the training period was the same?

86
Case Study 2 Anaconda

• 6 months and 840 generations later!!
• After 165 games it was rated at 2045.85 (sd
  33.94)
• Rated in the top 500 at zone.com (of the 120,000
  players now registered)
• That is better than 99.61 of the players

87
Case Study 2 Anaconda

• Playing Chinook8
• In a ten match series against Chinnok novice
  level it had two wins, two losses and 4 draws

8 Fogel D. B. and Chellapilla K. Verifying
Anacondas expert rating by competing against
Chinook experiments in co-evolving a neural
checkers player, Neurocomputing 42 (2002) 69-86
88
Case Study 2 Anaconda

• Blondie
• The neural checkers player went through a number
  of names
• David0111
• Anaconda
• Blondie24

89
Case Study 2 Anaconda
90
Case Study 2 Anaconda

• References
• Fogel D.B. Blondie24 Playing at the Edge of AI,
  Morgan Kaufmann, 2002
• Fogel D. B. and Chellapilla K. Verifying
  Anacondas expert rating by competing against
  Chinook experiments in co-evolving a neural
  checkers player, Neurocomputing 42 (2002) 69-86
• Chellapilla K and Fogel D. B. Evolving neural
  networks to play checkers without expert
  knowledge. IEEE Trans. Neural Networks
  10(6)1382-1391, 1999
• Chellapilla K and Fogel D. B.Evolution, neural
  networks, games, and intelligence, Proc. IEEE
  87(9)1471-1496. 1999
• Chellapilla K and Fogel D. B. Evolving an expert
  checkers playing program without relying on human
  expertise. IEEE Trans. Evolutionary Computation,
  2001
• Chellapilla K and Fogel D. B. Anaconda Defeats
  Hoyle 6-0 A Case Study Competing an Evolved
  Checkers Program Against Commercially Available
  Software. Proc. Of CEC 2000857-863