Title: TIELT: A Transfer Learning Evaluation Testbed for MultiAgent Adversarial Games
1TIELT A Transfer Learning Evaluation Testbed for
Multi-Agent Adversarial Games
David W. Aha Intelligent Decision Aids Group Navy
Center for Applied Research in AI Naval Research
Laboratory (Code 5515) Washington,
DC david.aha_at_nrl.navy.mil
Testbed for Integrating and Evaluating Learning
Techniques (FY04-FY05 IPTO Seedling)
The information presented in these slides reflect
the views of the author, not those of DARPA.
30 March 2005 TL Bidders Conference Arlington,
VA
2Context
Area A Transfer Learning Technology Development
- Area B Transfer Learning Evaluation Framework
- Testbed
- Experiment design and administration
3Outline
- Overview
- Objective
- Description
- Status Features, Collaborations, Application
- Transfer Learning
- Example challenge problem
- Future Plans
4TIELT Objective
Reduce the effort required to evaluate learning
techniques on performance tasks from complex
simulators.
Learning
5Learning in Cognitive Systems
Status
- Few deployed cognitive systems integrate
techniques that exhibit rapid enduring learning
behavior on complex tasks - Its costly to integrate evaluate embedded
learning techniques
6A Vision for Supporting Cognitive Learning
1986
7Goal Reduce Integration Costs
Problem Simulator/system integrations are
expensive! (time, )
1 integration
Decision Systemn
8Why Multi-agent Adversarial Games?
Attractive characteristics
- Several AI research challenges
- e.g., huge search spaces, adversarial,
collaborative, real-time (some), on-line,
uncertainty, multiple reasoning levels,
relational (social, temporal, spatial) data - Feasible Breakthroughs are expected to occur (w/
funding) - Inexpensive
- Military analog (i.e., training simulators)
- Popular (with public, industry, academia,
military)
9Gaming Genres of Interest(modified from (Laird
van Lent, 2001) (Fairclough et al., 2001))
10Real-Time Strategy Games (informal intro)
Goal Defeat opponents, build Wonder,
etc. Measure Score f(time, achievements) Map
Size (e.g., 128x128), location, terrain, climate
Time Stamp Real-time advancement
- State
- Map (partially observed)
- Resources
- Units
- Attributes Location, Cost, Speed, Weapon,
Health, - Actions Move, attack, gather, Build generator,
- Generators (e.g., buildings, farms)
- Attributes Location, cost, maintenance,
- Actions Build units, Conduct research, Achieve
next era - Consumables (used to build) Gold, wood, iron
- Research achievements Determines what can be
built - Actions selected
- Generators
- Units
- Diplomacy
- Adversaries State for each (partially observed)
11Wargus A Simple Decision Space Analysis
Variables
- W Number of workers
- A Number of possible actions
- P Average number of workplaces
- T Number of troops
- D Average number of directions a unit can move
- S Choice of troops stance
- B Number of buildings
- R Average number of research objectives
- C Average number of buildable units
Decision complexity per turn (for a simple game
state)
- O( 2W(AP) 2T(DS) B(RC) ) this is a
simplification - This example W4 A5 P3 T8 (offscreen)
D2 S3 B4 R1 C1 - Decision Space 1.4x103
- This becomes large with the number of units
- Motivates the use of domain knowledge (to
constrain this space) - Comparison Average decision space per chess move
is 30
12Multiplayer Adversarial Games for Transfer
Learning
Relevance
- Transfer learning allows humans to quickly
perform well - Significant background knowledge (e.g., score,
states, processes, objects, tasks, actions)
provides opportunities for rapid learning
13Target Audience Some Specifications
14TIELT Description
- A free, supported tool for integrating decision
systems with simulators - Users define their own APIs (message format and
content) - As-is license
- Initial focus Evaluating learning techniques in
complex games - Learning foci Task, Player/Opponent, or Game
Model - Targeted technique types Broad coverage
- Supervised/unsupervised, immediate/delayed
feedback, analytic, active/passive,
online/offline, direct/indirect,
automated/interactive
15Functional Architecture
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
Game Player(s)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
16Knowledge Bases
Game Interface Model
Defines communication processes with the game
engine
Decision System Interface Model
Defines communication processes with the decision
system
Game Model
- Defines interpretation of the game
- e.g., initial state, classes, operators,
behaviors (rules) - Behaviors could be used to provide constraints on
learning
Agent Description
Defines what decision tasks (if any) TIELT must
support
Experiment Methodology
Defines selected performance tasks (taken from
Game Model Description) and the experiment to
conduct
17UT Example Game Interface Model
Communication
Medium TCP/IP, Port 3000 Message Format ltnamegt
ltattr1gt ltvalue1gt ltattr2gt ltvalue2gt
Action Templates
TURN(Pitch real, Yaw real, Roll
real) SETWALK(Walk boolean) //Start walking or
running RUNTO(Target integer) //ID of object in
world
Sensor Templates
CWP(Weapon integer) //Change Weapon to Weapon
with this Id FLG(Id integer, Reachable boolean,
State Symbol ltheld, dropped, homegt
- Examples interface messages from the GameBots API
- http//www.planetunreal.com/gamebots/docapi.html
18UT Example Game Model
State Description
Operators
Players Array of Player Self Player Score
Integer
Shoot(Player) Preconditions Player.isVisible
Effects Player.Health - rand(10) MoveTo(Location
) Preconditions Location.isReachable()
Effects Self.position Location
Classes
Player Team String Number Integer
Position Location
Rules
GetShotBy(Player) Preconditions
Player.hasLineOfSight(Self) Effects
Self.Health - rand(10) EnemyMovements(Enemy,
Location1, Location2) Preconditions
Location2.isReachableFrom(Location1)
Enemy.position Location1 Effects
Enemy.position Location2
Location x Integer y Integer z
Integer
19UT Example Decision System Interface Model
Communication
Medium Standard I/O Message Format (ltnamegt
ltvalue1gt ltvalue2gt ltvalue3gt )
Example Template Messages Sent By TIELT
InitializeGameRules(ruleSet Array of
Rule) SendStateUpdates(CurrentState Array of
Object) LoadScenario(SavedGameFilename String)
Template Messages Received By TIELT
GiveAdvice(AdviceMessage String) PerformAction(Op
eratorName String, Parameters Array of
String) AskForValue(AttributeName String)
20UT Example Agent Description
Think-Act Cycle
Shoot Something
Pick up a Healthpack
Go Somewhere Else
Call Shoot Operator
Ask Decision System Where Do I Go?
Call Pickup Operator
Ask Decision System Where Do I Go?
21UT Example Experiment Methodology
Initialization
Game Model Unreal Tournament.xml Game Interface
GameBots.xml Decision System MyUTBot.xml Runs
100 Call slowdown(0.5)
Metrics
FragCount Self.kills FragsPerSecond
Self.kills/LengthOfGame AverageHealth
Self.health x
Plot
FragCount vs. Runs AverageHealth vs. of
players FragsPerSecond vs. Outdegree of net nodes
22Use Controlling a Game Character
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
23Use Prediction
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
24Use Game Model Revision
TIELTs User Interface
Evaluation Interface
Prediction Interface
Coordination Interface
Advice Interface
TIELT User
TIELTs Internal Communication Modules
Selected Game Engine
Selected Decision System
Learned Knowledge (inspectable)
TIELTs KB Editors
Selected/Developed Knowledge Bases
Game Model
Agent Description
Game Interface Model
Decision System Interface Model
Experiment Methodology
TIELT User
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
25A Researcher Use Case
- Define/store decision system interface model
- Select game simulator interface
- Select game model
- Select/define performance task(s)
- Define/select expt. methodology
- Run experiments
- Analyze displayed results
Selected/Developed Knowledge Bases
Knowledge Base Libraries
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
GM
AD
EM
GIM
DSIM
26Performance Tasks
TIELT Decision systems can be applied to
performance tasks that users select from the
simulator
Problem Solving Tasks
Analysis
Synthesis
Decision Support
Classification
Planning
Design
Scheduling
Diagnosis
Structural
Parametric
27An Example Performance Task
Task description
This involves several challenging learning tasks
Win a real-time strategy game
Subtasks and supporting operations
- Diagnosis Identify (computer and/or human)
opponent strategies goals - Classification Opponent recognition
- Recording Actions of opponents and their effects
- This repeatedly involves classification
- Diagnosis Identify goal(s) being solved by these
effects - Classification Identify goal(s), if solved, that
prevents opponent goals - Planning Select/adapt or create plan to achieve
goals and win the game - Classification Select top-level actions to
achieve goals - Iteratively identify necessary sub-goals and,
finally, primitive actions - Design (parametric) Identify good initial layout
of controllable assets - Execute plan
- Recording Collect measures of effectiveness, to
provide feedback - Planning If needed, re-plan, based on feedback,
at Step 2
28TIELT Status
Features (v0.6a)
- Pure Java implementation
- Message protocols
- Java Objects, Console I/O, TCP/IP, UDP
- Note Now supports external shared memory
- Message content User-configurable
- Instantiated templates tell it how to communicate
with other modules - Messages
- Start, Stop, Load Scenario, Set Speed,
- Agent description representations (w/ Lehigh
University) - Task-Method-Knowledge (TMK) process models
- Hierarchical Task Networks (HTNs)
- Experimental results can be stored in databases
supporting JDBC or ODBC standards (e.g., Oracle,
MySQL, MS Excel)
29TIELT http//nrlsat.ittid.com
30Current Development Priorities
- Speed Compile TIELT Script XML to Java
- GUI
- Parse interface code from free text (e.g., emacs
plugin) - Simplify deletion of variables
- Graphical depiction of system information flow
- Language
- TSL Higher level data structures (e.g., Vector,
Hash) - Game Model Parameterized game events
- State Separate agent state from game state
- Experiment Methodology
- More analytical tools
- Link independent variables with the other models
- Debugging
- Real-time view of the Game Model
- More powerful logging scheme (e.g., access to log
levels)
31FY05 Collaboration Projects
User Interfaces
TIELT User
Prediction Interface
Evaluation Interface
Coordination Interface
Advice Interface
U.Minn-D.
USC/ICT
U.Mich.
Game Library
TIELT Test Bed
Decision System Library
ISLE ICARUS Lehigh Case-based
planner U.Michigan SOAR UTA DCA UT Austin
Neuroevolution
ToEE
EE2
Mad Doc
Troika
FreeCiv
NWU
ISLE
KB Editors
Knowledge Base Library
LU, USC
Mich/ISLE
U. Mich.
Many
Game Model
Agent Descriptions
Game Interface Model
Decision System Interface Model
Experiment Methodology
Many
TIELT User
32TIELT APIs Status
33Collaborator Lehigh University
PI Héctor Muñoz-Avila
- Integrate TIELT with Stratagus/Wargus (i.e., to
control an AI player)
Actions (TCP/IP)
STRATAGUS Modified Client
TIELT
Sensors (TCP/IP)
34First Research Application (w/ Lehigh U.)
Previous work context
Ponsen, M.J.V., Muñoz-Avila, H., Spronck, P.,
Aha, D.W. (IAAI05). Automatically acquiring
adaptive real-time strategy game opponents using
evolutionary learning.
Tactics1
Tactics1
Game Opponent Scripts
Evolved Counter- Strategies
Dynamic Scripting
Genetic alg.
. . .
. . .
Tactics20
Tactics20
Initial Policy (i.e., equal weights)
Opponent-Specific Policy f(Statei)Tacticij
Action Language
- The tactics (action sequences) you can execute
depends on which units and technology is
available, which depend on which buildings youve
constructed. - Thus, the 20 building states in Wargus are
manually predefined and correspond to the set of
buildings a player possesses.
Wargus
35First Research Application (w/ Lehigh U.)
Our contribution CaT
Aha, D.W., Molineaux, M., Ponsen, M.
(ICCBR05). Learning to win Case-based plan
selection in a real-time strategy game.
Learns to Select Tactics fi(Casesi, featuresq)
tacticij
Acquired Cases (1 set per Building
State) caseijltBuilding statei, featuresj,
tacticij, performancegt
Hypothesis CaT can significantly outperform
(i.e., higher winning percentage and higher
average relative scores) the evolved
counter-strategies
Evolved Counter- Strategies
Evaluation Interface
Matt Marc
Game Opponent Scripts
TIELTs Internal Communication Modules
Editors
Tactics1
Wargus
CaT
. . .
Matt
Tactics20
Knowledge Bases
(Case-based Tactician)
36First Research Application (w/ Lehigh U.)
Our results (LOOCV)
We relaxed the assumption that the
opponent/adversary is static
37Collaborator Mad Doc Software (MDS)
PI Ian Lane Davis
- Integrating state-of-the-art real-time strategy
game engines with TIELT
Option 1 Commercially available Empire Earth II
engine (w/ patch)
Option 2 Custom TIELT-enabled Mad 3d engine
(MOUT focus). (Note EE II was built using the
Mad 3d engine.)
- Challenge problems (for TIELT competitions at
workshops)
38Collaborator University of Michigan
PI John Laird
- Simple game integration (Towers of Hanoi) to
familiarize with TIELT - SOAR integration
SWIG Wrapper
SML
Game-specific
Game/Sim
Game Interface Model
Kernel SML/ Soar
Agent Description
Actions
Library calls and returns
ClientSML
Slots
Decision System Interface Model
Library calls and returns
Sensors
SML
General
- SML (Soar Markup Language) is an in-house XML
wrapper around UMs gSKI-Soar interface - SWIG is a wrapper that provides access to their
C SML functions (from any prog. language) - Slots are places to store data that is shared
between a decision system and TIELT
- SOAR application (e.g., Mad Docs game engine)
39Collaborator USC/ICT
PI Michael van Lent
- Simple game integration (Tic-Tac-Toe) to
familiarize with TIELT - Integration of Full Spectrum Command/Research
(FSC/R) with TIELT - Note FSC 1.5 is a PC-based training aid that
models the command and control of a U.S. Army
Light Infantry Company in a MOUT environment. - Design completed
- Initial API has been delivered includes a subset
of sensors and actions - General knowledge bases (KBs)
- A general game-play KB for knowledge common to a
game genre - 2 game-specific KBs that, with (a), can be used
to play 2 games in the genre
402. TIELT Support for Transfer Learning
Goal
Show that comprehensive knowledge learned from
experience with a complex game can be transferred
to increase initial performance, performance
rate, or asymptotic achievable performance
Cognitive System
Learning Module
Transferring learned knowledge
Learned Knowledge
What can be transferred?
- Experiences
- Plans for achieving goals
- e.g., for situation assessment, adversary intent
recognition, course-of-action generation and
analysis, and ultimately to win a game - User preferences
41City Control Challenge Problem (MDS)
Real-time strategy game, where the goal is to
gain control over city blocks that vary in
utility, and require different levels of control.
Design due 4/15, implementation due 5/15, in
time for summer workshops.
42Initial Challenge Problem Components
City Control A Planning Challenge Problem
- State S
- Map M B11,,Bmn A map
consists of a set of city blocks - City Block Bij
- features(Bij) f1(),,fl() A block
has a set of features (e.g., location, utility) - objects(Bij)?O
- Objects O objectType(), location(), status(),
- Units U player(), type(), actions(), weapon(),
armor(), - Resources R type(), amount(),
- Categories Controllable (e.g., petrol,
electricity), natural (e.g., uranium, water) - Actions A
- apply(Ui,Aj,S) S?
Units can apply actions to modify the state - Tasks T type(), preconditions(), subtasks(),
subtaskOrdering(), ... - Categories Strategic, tactical (where primitive
tasks invoke actions)
- CP M,S0,SG,F,R,E
- M Map
- SG Goal is to achieve a state (i.e., control
of specified city blocks) - e.g., (Bij, Gij) Bij?M Gij?G (1im, 1jn)
- Goals can be achieved by tasks or actions
- F Friendly units
Controllable units - R Resources Objects
that can be used in actions - E Enemy units
Non-controllable adversary units
43Example Challenge Problem (Sketch)
City Control A Planning Challenge Problem
- Goal Control of a set of city blocks BG
currently under control of E - Starting State
- F all located in on block Bij (of varying
significance) - 3 concentrations of Ei
- Notes
- Success may involve controlling some
strategic/tactical blocks BST ? BG - Securing blocks nearby BG may assist with the
goal - Controlling a block Bij involves leaving F??F on
Bij - Several issues involved with force distribution
decisions
Block structure of the challenge problem map
44Example CPs for the Transfer Learning Levels
Differing
Train on a turn-based game, test on a real-time
game
Reformulating
Train on one real-time game, test on another
Train when using deception only for unit
locations, and test when deception is also used
for weapon identities
Generalizing
Abstracting
Vary weapons and armor available
Train with only dismounted or only mounted
soldiers, test with them both available
Composing
Restyling
Vary map M
Vary number of units in F and/or E
Extending
Vary non-combatants on M
Restructuring
Change composition of F and/or E
Extrapolating
Change initial locations for units in F and/or E
Parameterization
Memorization
Same S0 and SG
45Example Transfer Learning Problems
Extending
Wargus 3x1 battle
Wargus 1x1 battle
Reformulating
46Some Experimental Variables
Dependent variables
- Goal(s) achievement frequency
- Solution quality (where applicable)
- Differences from SG
- Time/moves required
- Losses
- Units
- Resources expended
- Control
Independent variables
- Transfer learning level
- Game(s)
- Initial state
- Amount of applicable domain knowledge
47Some Issues for Defining CPs
- Judicious domain knowledge can decrease problem
complexity - e.g., win Wargus given complete plans, state
lattice (Ponsen Spronck, GAME-ON04) - Thus, domain knowledge must be controlled, or
measured, to compare TL approaches
- Some problem characteristics can apply at all
transfer learning levels - e.g., Percent observability of opponent weaponry
and/or model - Thus, problem definitions should hold their
values constant (if feasible) to isolate the
impact from the independent variables of interest
(i.e., those that define the TL level)
- Goals can appear at different transfer levels
- e.g., Select building location, with a partial
(Restyling) vs. the complete map (Memorization) - Thus, higher transfer levels dont imply that the
goals must be more ambitious
48AAAI05 General Game Playing Competition(M.
Genesereth, Stanford U.)
3. Future Plans
Goal Win any game in a pre-defined category
- Initial category Chess-like games
- Games are produced by a game generator
- Input Rules on how to play the game
- Move grammar is used to communicate actions
- Output (desired) A winning playing strategy
Annual AAAI Competition
- WWW games.stanford.edu
- AAAI05 Prize 10K
49Games
Dimensions One player or n players
Simultaneous or alternating play x Complete or
incomplete information x Competition and
Cooperation Current Examples
Blocks Minichess Buttons and
Lights Endgame Maze Corridor
Tictactoe Roshambo Ticticoe Diplomacy
50Game Description Language
Game Description Language (GDL) Conceptual
Basis - Relational Nets Logical Encoding
using Fluent Representation Specification
of GDL published Issue Currently complete
information games only Future partial
information games as well
51GGP-TIELT Integration
Use Case
- Researcher develops a General Game Player
competitor - Connects to GGP Game Manager through TIELT
- Using TIELT, designs experiment for playing n
matches of m different General Games, against
reference opponents and unknown GGP competitors - Runs experiment online through GGP Game Manager
- Evaluates experimental results using TIELT
evaluation methodologies and GGP performance data
available online
52TIELT as Intermediary
GGP Reference Player
GGP Game Manager
TIELT
- Parse GGP
- messages
- Store in KBs
- Reform messages
- from KB and
- pass to player
- Relay moves
- back to GGP GM
Runs as normal TIELT interface of player is
transparent to Game Manager.
Currently, a GGP player running at Stanford
in general, any TIELT-capable or GGP player.
53Upcoming TIELT-Related Events
General Game Playing Competition
- AAAI05 (24-28 July 2005 Pittsburgh)
- Chair Michael Genesereth (Stanford U.)
- We our integrating TIELT with GGP, attracting
additional competitors
TIELT-Related Workshops
- At IJCAI05 (31 July 2005 Edinburgh)
- Reasoning, Representation, and Learning in Gaming
- Co-chairs Héctor Muñoz-Avila, Michael van Lent
- At ICML05 (7 August 2005 Bonn)
- Knowledge-Intensive Learning in Simulated Task
Environments - Co-chair Dan Shapiro
- At ICCBR05 (24 August 2005 Chicago)
- Computer Gaming and Simulation Environments
- Co-chair David C. Wilson
54TIELT Support for Transfer Learning
Project Description
- Key Ideas TIELT enables the evaluation of
transfer learning techniques on complex simulated
gaming tasks. - Facilitating Breakthroughs
- Configurable Interfaces For both simulators
cognitive systems, these simplify their
integration with TIELT. - Interface libraries Interfaces for
previously-integrated simulators simplify access
to challenge problem tasks, while interfaces for
previously-integrated cognitive systems permit
immediate comparison with competing baseline
approaches. - Transfer opportunities Complex games provide
many opportunities for transfer learning at all
levels.
TIELT facilitates evaluations of methods that
transfer declarative and procedural knowledge
among gaming tasks.
Payoff
Technical Details
Goal Encourage researchers to evaluate their
transfer learning techniques on challenge
problems concerning tasks defined in complex
simulators (e.g., games).
Challenge Problems TIELT will supply a highly
flexible environment for defining challenge
problem tasks on transfer learning, linking to
simulators that can execute those tasks, and
testing approaches on those tasks.
- TIELT Features
- Communication Supports several key message
protocols (TCP/IP, Java objects, console I/O,
UDP), and message content is user-configurable. - Game representation Multiple formats (e.g., TMK,
HTN) are supported for declarative procedural
game knowledge. - Task selection Permits cognitive system to focus
on user-selected subtasks in a task network, and
creation of agents to solve the remaining tasks. - Testing User-defined evaluation methodology.
- Java implementation Works on many software
platforms.
Competition Vision TIELT, challenge problem
definitions, and interfaces with required
simulators and competing cognitive systems will
be made available through the TIELT www site.
Documentation (e.g., tutorials) and on-line
support for using TIELT will address all user
questions. Releases of training and testing
problems will be done according to the PMs
schedule. TIELT will be used to evaluate each
approach.
55Backups
56TIELTs Internal Communication Modules
Database
Evaluation Interface
Advice Interface
Database Engine
State
Evaluator
Controller
Stored State
Current State
Translated Model (Subset)
Decision Communicator
Model Updater
Selected Decision System
Learning Task
Selected Game Engine
Percepts
Action / Control Translator (Mapper)
Decision System Outputs
Actions
Perf. Task
Game Model
Game Interface Model
Agent Description
Decision System Interface Model
Experiment Methodology
User
Game Interface Model Editor
Decision System Interface Model Editor
Game Model Editor
Agent Descr. Editor
Expt. Method. Editor
57Sensing the Game State (City placement example,
inspired by Alpha Centauri, etc.)
1
In Game Engine, the game begins a colony pod is
created and placed.
TIELT
Current State
2
The Game Engine sends a See sensor message
identifying the pods location.
4
Action Translator
Actions
3
Updates
Game Engine
1
The Model Updater receives the sensor message and
finds the corresponding message template in the
Game Interface Model.
Sensors
3
5
2
Model Updater
Controller
3
4
Game Model
4
Game Interface Model
This message template provides updates
(instructions) to the Current State, telling it
that there is a pod at the location See describes.
User
Game Interface Model Editor
Game Model Editor
5
The Model Updater notifies the Controller that
the See action event has occurred.
58Fetching Decisions from the Decision System (City
placement example)
1
The Controller notifies the Decision Communicator
that it has received a See message.
TIELT
Controller
2
The Decision Communicator finds a city location
task, which is triggered by the See message. It
queries the controller for the learning mode,
then creates a TestInput message to send to the
decision system with information on the pods
location and the map from the Current State.
Selected Decision System
1
Decision System Outputs
Action Translator
4
Translated Model (Subset)
Learning Module 1
Decision Communicator
Current State
3
. . .
2
2
Learning Module n
Agent Description
Decision System Interface Model
3
The Decision Communicator transmits the TestInput
message to the Decision System.
User
4
The Decision System transmits its output(s) to
the Action Translator.
Decision System Interface Model Editor
Agent Desc. Editor
59Acting in the Game World (City placement example)
1
The Action Translator receives a TestOutput
message from the Decision System.
4.b, c
The Advice Interface receives Move and displays
advice to a human player on what to do next, or
makes a Prediction.
2
The Action Translator finds the TestOutput
message template, determines it is associated
with the city location task, and builds a MovePod
operator (defined by the Current State) with the
parameters of TestOutput.
TIELT
Advice Interface
Prediction Interface
4.b
4.c
Current State
1
2
Actions
Action Translator
4.a
Game Engine
3
3
The Action Translator determines that the Move
Action from the Game Interface Model is triggered
by the MovePod Operator and binds Move using
information from MovePod.
2
3
Game Interface Model
Decision System Interface Model
User
Game Interface Model Editor
Decision System Interface Model Editor
4.a
The Game Engine receives Move and updates the
game to move the pod toward its destination, or
60Promising Learning Techniques
61We solicit and fix bugs via this site