Automatic Construction of Static Evaluation Functions for Computer Game Players

1
Automatic Construction of Static Evaluation
Functions for Computer Game Players

• D1 Miwa Makoto
• Chikayama Taura Lab

2
Table of Contents

• Introduction
• Computer Game Player
• Static Evaluation Functions
• Related Work
• Automatic Construction of Static Evaluation
  Functions
• GLEM
• Proposal
• Experimental setting and result
• Summary and future plan

3
Computer Game Player

• Game
• 2-player, zero-sum, deterministic, and perfect
  information game
• ex. chess, Shogi, Go, etc
• Computer Game Player
• receives a position and returns the next move
• Important Features
• Evaluation Functions
• Game Tree Search ( search strategy)

4
Static Evaluation Functions (SEFs)

• Static ? without search
• SEFs evaluate positions and assign heuristic
  values to the positions
• The values rank the positions
• ?Static evaluation functions decide the computer
  game players behavior

5
Construction of SEFs

• To write SEFs by hand, developers need ...
• Deep knowledge about the game
• To find useful features and essential features
• Large amount of effort to tune
• ex. self-play, play with other players
• What if no experts exist?
• What if they have no time to tune?
• What if they want to guarantee that the SEFs are
  optimal?

6
Automatic Construction of SEFs

• Construction from simple features
• Simple features
• A set of features to represent a input position
• Match results
• 2 kinds of methods
• Direct Method
• Genetic Programming, Neural networks, etc
• Hybrid Method

7
Hybrid Method

• Linear combination of high-level features from
  simple features
• Feature (in comparison with direct method)
• Less expressive
• Lower generation and running costs
• Easy to analyze and optimize constructed SEFs by
  hand
• ?Fast SEFs
• Ex.
• ZENITH, GLEM

8
GLEM (M. Buro, 1998)

• Generalized Linear Evaluation Model
• Application Othello
• Logistello
• One of the strongest computer Othello players
• won the human champion in 1997

9
GLEM

• Extraction of atomic features by hand
• (as simple as Is a black King on A1?)
• Generation of configurations
• conjunctions of atomic features
• Selection of frequent conjunctions in training
  positions
• Weighting configurations using linear regression

10
Pattern

• Treating a whole position at once costs high
• Extract a part of the position called pattern by
  hand
• Pattern values are pre-computed and preserved in
  a hash table
• ? Fast Evaluation

Evaluate patterns using a hash table
11
GLEM

• Notable success only in Othello
• No other applications
• No general way to select patterns
• Selection of frequent conjunctions
• No theoretical explanation of the relationship
  between frequent conjunctions and important
  features

12
Proposal

• Hybrid Method
• Construct evaluation functions from simple
  features using frequency and conditional mutual
  information

13
Expression of Evaluation Function

• Linear combination of evaluation features
• Low generation and running costs
• Easy to analyze and hand-optimize constructed SEFs

14
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

15
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

16
Basic Features

• Binary
• A set of features to represent the inputs
• Minimum evaluation features
• ex. O is on A1 in tic-tac-toe
• Not simple features
• ex. Three Os are in the 1st row in tic-tac-toe

17
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

18
Pattern

• Conjunction of basic features
• 2basic features patterns
• ? Need selection
• An evaluation feature is a pattern

19
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

20
Prior Selection Based on Frequency
Train positions 1,2,5,6,7,9 2,3,4,5 1,2,7,8,
9 1,7,9 2,7,9 2
Frequent closed patterns 1, 7, 9 2, 7, 9 7,
9 2
Frequent patterns 1 1,7 1,9 1,7,9 2,7
2,9 2,7,9 7 9 7,9 2 minimum support
3
21
Prior Selection Based on Frequency

• Frequent pattern
• A pattern that appears in more than x positions
• x minimum support
• Closed pattern
• A maximal element of patterns which appear in the
  exactly same positions
• Extraction of frequent closed patterns
• LCM (Linear time Closed set Miner)
• H. Arimura et al., 2005

22
Prior Selection Based on Frequency

• Selection of frequent closed patterns
• Select frequent patterns
• To avoid overfitting
• Evaluating all patterns is impossible
• Ignored infrequent important patterns may exist
• Select closed patterns
• To reduce the cost of CMIM selection
• Not closed patterns will be deselected

23
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

24
Select Suitable Patterns

• Not all frequent closed patterns are important
  features
• ?Need selection
• There are a large number of frequent closed
  patterns
• ?Selection by evaluating all frequent closed
  patterns at once costs high
• ?CMIM

25
CMIM (Conditional Mutual Information
Maximization) F. Fleuret, 2004

• Purpose
• Select both individually informative and
  two-by-two weakly dependant patterns
• Method
• Select a pattern which has a large conditional
  mutual information for every selected pattern

26
Conditional Mutual Information

• Entropy
• Randomness of a probability variable
• Mutual information
• s information about
• Conditional mutual information
• New s information about after obtaining
  s information about

27
Select Suitable Patterns

• It is difficult to apply CMIM selection to a
  number of frequent closed patterns
• Top-k selection
• Select top k patterns from each position based on
  mutual information between the patterns and the
  labels before CMIM selection

28
Steps of the Proposed Method

• Extraction of basic features by hand
• Generation of evaluation features
• Prior selection of frequent closed patterns in
  training positions
• Selection of suitable patterns using conditional
  mutual information
• Weighting evaluation features using a Naïve
  Bayesian classifier

29
Naïve Bayesian Classifier

• A simple likelihood ratio test with an assumption
  of conditional independence among features

30
Experimental Setting

• Application
• Mate problem in shogi
• Target Evaluation Function
• returns a prediction value which indicates how
  likely the enemy king is to be mated
• Basic Features
• Positions and effects of pieces
• 37,336 features

31
Experimental Setting

• 89,768 positions (24,945 mated)
• 80,000 train and 9,768 test
• Extract randomly from those positions visited
  within search from endgame positions
• Labeled using PDS within the limit of 5,000,000
  nodes
• Experimental environment
• Intel Xeon 3.06GHz dual, 2GB RAM
• AMD Opteron 840 (1.4GHz) 4, 4GB RAM
• AMD Opteron 250 (2.4GHz) 2, 8GB RAM
• Language C

32
Prior Selection Based on Frequency
minimum support 2 (1600)
33
Prior Selection Based on Frequency

• When the minimum support is 2 of all training
  positions,
• it takes about 38 minutes to select 38,173,197
  patterns on Xeon
• about 88 redundant patterns are eliminated by
  selecting closed patterns only
• the longest pattern length is 23

34
Select Suitable Patterns
top-k selection (minumum support 5, 10,000
positions)
35
Weighting Evaluation Features
As for k500, 1000, we divided training positions
36
Select Suitable Patterns Weighting Evaluation
Features

• CMIM selection weighting
• 74 in accuracy
• (k200, 2,000 evaluation features)
• About 7 worse than a evaluation function with
  evaluation features selected by hand
• As for k500, 1000, selection using divided
  training positions caused worse results

37
Summary

• New method to generate evaluation functions
  automatically
• An application to the mate problem in shogi
• Evaluation function with 74 in classification
  accuracy

38
Future Plan

• Reduce cost of selection
• General extraction of GLEMs patterns
• This enables to try smarter selection and more
  expressive conjunction
• Approximate methods
• Simulated annealing, beam search, etc
• Apply to other games
• ex. Othello, Go, etc
• To show that our method is applicable to other
  games
• Compare with the other methods

39
Frequent Closed Pattern
blank
40
Frequent Closed Pattern
blank