Explaining Soar: Efforts to make Soar models more useful and usable

1
Explaining SoarEfforts to make Soar models more
useful and usable

• Isaac G. Councill
• Work conducted with Frank E. Ritter, Steven R.
  Haynes, and Mark Cohen
• This project is supported by the US Office of
  Navy Research, award number N00014-02-1-0021

2
Overview

• Brief introduction to Soar
• Problem statement
• User study (ICCM 2003)
• dTank sim (Tech. Report No. ACS 2003 - 1 )
• Herbal Viewer
• Herbal IDE (23rd Soar Workshop)

3
Soar

• Cognitive architecture / AI programming language
• Production system

4
What is Soar used for?

• Intelligent synthetic forces
• Special ops training
• Fixed/Rotary aircraft scenarios
• Sensitivity training

• Until recently, JSAF was largest training
  environment
• Hundreds of TacAir soar agents participated with
  thousands of agent world wide in massive
  simulations

5
The ProblemBehavior Validation

• Lives are on the line improper training leads
  to mistakes in the field
• Training personnel must manage 40-100 entities
  and ensure proper training experience
• Complex agent behavior is hard to understand
• Best interfaces not good enough (TacAir Soar
  discontinued in training exercises partly for
  usability reasons)

6
User Study
7
Motivation

• Easier validation of agent behavior
• Increased usability
• Agents become easier to understand - ala Mycin
• Provide better training - ala Clancey (Neomycin)
• Increased potential to develop users trust
• Enhanced understanding may enable more creative
  use of tools

8
TacAir-Soar SAP

• Allows users to access and interpret internal
  state of agents
• Supports limited view of agent operating history
  at run time, full off-line replay capability
• Provides what, users must infer why

9
SAP Usability Data (Avraamides Ritter 2002)

• Expert SAP users observed while performing a set
  of basic SAP tasks
• Participants encouraged to provide verbal
  explanations of agent behavior and interpret
  scenarios displayed by SAP
• 4 hours of video recordings of expert
  TacAir-Soar SAP users interacting with the SAP

10
Study Method

• Videos transcribed and analyzed to determine
  where breakdowns occur in users attempts to
  understand TacAir-Soar behavior
• Model/View/Controller used as framework for
  analysis.
• Model ? explanation content
• View ? delivery mechanisms

11
Results Explanatory content

• Need for explanations (14, 9) validates
  premise
• Request for additional content (10, 5)
• Beyond model scope (5, 3)
• Within model scope (5, 2)

12
Results Delivery

• Milestones (6, 2)
• Clarity (2, 1), Content (4, 2)
• Goal stack (12, 8) disliked
• Clarity (2, 2), Content (10, 6)
• Amplified goal display (6, 3) new to most
  participants
• Proposal for new delivery tool (6, 8)

13
Results Delivery (viewport)

• 56 utterances, 36 of transcripts
• Scale orientation (8, 8)
• User ergonomics (6, 4)
• Scale as awareness (2, 4)
• Supplemental information (22, 11)
• Usefulness of information (16, 7)
• Requests for additional info (6, 4)
• Symbols (26, 18)
• Usefulness of symbols (12, 7)
• Requests for additional symbols (14, 11)

14
Deriving Explanations

• A major use for SAP in applied context is to
  support derivation of explanations for agent
  behavior
• Explanation derivation is currently interpretive,
  lacking explicit support
• Viewport is most used tool for deriving
  explanations

15
Process Information

• Goal stack is only conveyor of process
  information
• Found to be useless by half of participants, of
  marginal utility by other half
• All participants indicated incomplete
  understanding of goal stack
• All process knowledge must be inferred by users
  based upon assumptions about TacAir Soar
  implementation

16
Understanding the viewport

• Participants indicated confusion as to purpose of
  viewport agent knowledge or real world?
• Viewport displays selected declarative knowledge
  (but look how much)
• Exploits unrealistic aspects of model
• As we develop realistic models of working memory
  and attention, viewport-like tools may become
  less useful

17
General Conclusions

• Efforts should focus on general mechanism for
  accessing agent knowledge and contrasting the
  knowledge with real world
• Information regarding the process of agent
  cognition should be made more accessible

18
dTankDistributed Agents in a Competitive
Environment
19
Overview

• Adversarial test bed for agents and agent
  facilities
• Provides an agent architecture-neutral interface
  to the game server
• Humans and artificial entities built on virtually
  any platform can interact within the same
  environment over a LAN or the internet
• Highly dynamic API!

20
dTank Server Interface
21
Herbal Viewer
22
Purpose

• Herbal makes it easier to understand running
  models of Soar
• Visual displays of working memory, and the PSCM
  task structure.
• Dynamic trace views that track the history and
  frequency of events.
• Displays that profile a running model.

23
Herbal is Flexible

• Herbal will work with all Soar models. Nothing
  has to be done to the model for it to work with
  Herbal!
• Herbal allows for the replay of previous model
  sessions without the need to run Soar.
• Although currently designed to work with Soar,
  Herbal can be extended to work with many
  cognitive architectures, as long as the
  architecture is based on the PSCM.

24
Architecture
Herbal
VISTA
Soar
Tcl Script
Soar Monitors
Soar Events

• Herbal
• Written in Java.
• Uses the VISTA library to listen for event
  strings over a TCP/IP socket.
• Soar Monitors
• A collection of Soar event monitors (productions)
  that fire on queue and call functions located in
  the Tcl Script.
• Tcl Script
• Responds to Soar events by sending event strings
  to Herbal, over a socket.

25
VISTA

• The Visualization Toolkit for Agents (VISTA),
  developed by Soar Technology Inc.
• Facilitates the creation of agent visualization
  applications by providing an infrastructure for
  communication between agents and VISTA enabled
  applications.
• Using a communication channel, agents can convey
  changes to their internal state to a listening
  VISTA enabled application.
• The VISTA toolkit also provides the ability to
  record and playback agent activity.
• By using VISTA as the infrastructure for
  communication between the cognitive architecture
  and Herbal, a significant amount of development
  time was saved.
• VISTA Developers Handbook, Soar Technology
  Inc., 2002.

26
The Tree and Graph View
27
The History and Frequency Traces
28
The Commentary and Command View
29
Herbal IDE
30
Overview

• Protégé
• An objectification of Soar the PSCM, roughly
• Merging the design and programming processes
• Aspect-oriented software development
• Herbal IDE architecture summary

31
Protégé

• Ontology editor used to create object systems
  that describe all concepts and concept relations
  within a domain.
• There are many plugins for protégé, including
  support for xml and rdf output.

Protégé is a free download from
http//protege.stanford.edu
32
An Objectification of Soar

• Specified using Protégé
• Constructed using PSCM as base model, with
  augmentations deemed appropriate
• Encourages top-down design and programming

33
Merging Programming and Design
34
In effect
35
Aspect-Oriented Programming

• One step beyond OOP
• Programmatic behaviors often do not fit naturally
  discrete modules (e.g. design patterns, logging,
  optimization levels)
• AOP allows modularization of cross-cutting
  behaviors in object systems

36
AOP Example

• Common solution to logging
• public void doGet(JspImplicitObjects theObjects)
  throws ServletException
• logger.entry("doGet(...)")
• JspTestController controller new
  JspTestController() controller.handleRequest(theO
  bjects) logger.exit("doGet")

37
AOP Example (cont.)

• Create an aspect to objectify logging and avoid
  repetitive coding
• public aspect AutoLog
• pointcut publicMethods() execution(public
  org.apache.cactus..(..))
• pointcut logObjectCalls() execution(
  Logger.(..))
• pointcut loggableCalls() publicMethods()
  ! logObjectCalls()
• before() loggableCalls()
• Logger.entry(thisJoinPoint.getSignature().t
  oString())
• after() loggableCalls()
• Logger.exit(thisJoinPoint.getSignature().to
  String())

38
AOP in the Herbal IDE

• Soar is difficult to program/understand/explain!
• One way to slice it The interesting parts of
  models arent just the operators they have in
  them, but how those operators interact with each
  other (aspects!)
• Flexible aspect creation allows developers to
  specify reusable behaviors
• More principled design
• Better encapsulation of interesting bits
• More understandable and explainable models
  (thats the main point, afterall)

39
Herbal IDE Architecture
40
Resources

• Protégé http//protege.stanford.edu
• RDF http//www.dlib.org/dlib/may98/miller/05mill
  er.html
• XSLT http//www.xml.com/pub/a/2000/08/holman/
• Aspect-oriented programming
• http//www.parc.xerox.com/aop
• http//aosd.net