Verification, Validation and Accreditation of AgentBased Simulations - PowerPoint PPT Presentation

Loading...

PPT – Verification, Validation and Accreditation of AgentBased Simulations PowerPoint presentation | free to download - id: 327b6-OGVlM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Verification, Validation and Accreditation of AgentBased Simulations

Description:

An ABS is a simulation in which entities have 'agency' ... Why some Computational Social Scientists prefer ABS ... we expect from a theoretically perfect ABS? ... – PowerPoint PPT presentation

Number of Views:75
Avg rating:3.0/5.0
Slides: 18
Provided by: pae79
Learn more at: http://www.scs.gmu.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Verification, Validation and Accreditation of AgentBased Simulations


1
Verification, Validation and Accreditation of
Agent-Based Simulations
  • Deborah Duong

2
Purpose
  • To introduce Agent-Based Simulation
  • To propose measures of effectiveness for
    Agent-Based Simulation

3
What is an Agent-Based Simulation?
  • Agent-Based Simulation (ABS) is broadly defined
  • An ABS is a simulation in which entities have
    agency
  • Agents can perceive and behave in their
    environment based on goals
  • Agent-Based Simulation is used for modeling
    living systems
  • Biological and social systems
  • Non-living systems are mindless, and therefore
    dont have agency
  • The concept of emergence is important
  • Agents behave according to one set of rules
  • New patterns emerge from individual behaviors
  • Emergence is micro-macro integration

4
How does Agent-Based Simulation Compare ?
  • Other methods that dont involve agency or minds
    are also used to describe living systems
  • Discrete Event Simulation
  • Events of a process are scheduled to occur at
    discrete points
  • System Dynamics Simulation
  • Looks at the flow of fluid levels over time
  • Time delays are important
  • Social Networks
  • Patterns in the arrangement of entities to each
    other are important
  • These methods are at their best when modeling
    non-mental phenomena
  • Ecology
  • Predator-Prey cycles
  • The Economy
  • Cycles not based on beliefs (like the stock
    market is)
  • Any time entities act similarly
  • Everybody eats!
  • Non-agent simulation methods model flows and
    arrangements of averaged entities
  • Their State does not change, because entities
    are not modeled explicitly
  • They are not networked
  • They are viewed from an external, etic
    standpoint

5
Why some Computational Social Scientists prefer
ABS
  • Their preference depends on their feelings on the
    importance of agency and minds
  • They may believe that other tools are not as rich
  • Other tools tend to make heroic assumptions
  • They often can not model the crux of the problem
  • They are more descriptive than causal
  • North and Macal
  • We believe that in the future virtually all
    computer simulations will be agent-based because
    of the naturalness of the agent representation
    and the close similarity of agent models to the
    predominant computational paradigm of
    object-oriented programming.

6
Cognitive vs. Reactive Agents
7
Data-Based vs. Theory-Based ABS
8
Agent Based Simulation and VVA
  • Verification
  • Determination of whether a simulation expresses a
    theory well
  • Validation
  • Determination of whether a simulation has
    fidelity with the real world
  • Accreditation
  • Determination that a simulation is useful for
    analysis of a particular domain
  • Verification, Validation and Accreditation of
    agent based models is problematic
  • VVA originated in physics models
  • The nature of social science has implications for
    agent based VVA

9
Agent-Based Simulation and Verification
  • The more a simulation has the power to express a
    theory, the more the simulation is verified
  • A System Dynamics model of a verbal theory
    wouldnt have a high degree of verification
    unless that theory was about time-delays
  • The referent of any mathematical or simulation
    model is a theory
  • In physics based models, verification is doable
  • In physics-based models, verification is mainly
    about bugs
  • Replication, or using a different method to
    simulate the same theory, can help debug agent
    based social models
  • In social-science based agent models,
    verification is the central issue
  • Verification is about technology to represent an
    idea
  • Newton had the technology of the calculus
  • The technology to simulate social theories is not
    trivial
  • For example, a social theory about human learning
  • may need a computer that can match a human in
    learning
  • With knowledge of available tools and creativity,
  • Verification is just a matter of good
    (scientific) taste, for now

10
The Social Literature as the Referent
  • Fitting raw data is not enough for verification
  • Data can be over-fitted
  • One could simulate by never addressing cause,
    by only making correlated things appear magically
  • Since why is not modeled, the simulation is not
    generally applicable
  • If it wont model a new situation, it wont model
    itself well either
  • If there are no causes a level under the
    phenomena you model, you are only describing, not
    analyzing
  • You can not explore the new levers to change
    outcomes, other than the ones you put in the
    simulation to begin with
  • Data should be fitted through a theory of social
    science
  • Thoughtful models in the social literature are
    preferred to models from other fields
  • Just because we have the tools to describe time
    delays, physical phenomena, and epidemiology
    doesnt relate them to social theory
  • Knowledge of all tools is needed to model the
    richness of the social world
  • Tested by surveying the relative frequency of
    issues in the social literature and comparing to
    the relative frequency of issues in an ABS

11
Agent-Based Simulation and Validation
  • The more explanatory power an agent-based
    simulation has, the more the simulation is
    validated
  • A simulation model should match the data in the
    world in the way that its theory matches it
  • Validation of agent based simulation is dependant
    on verification If an agent based simulation is
    not first verified, it will not be valid
  • Validation of agent based simulation is dependant
    on the explanatory power of its referent theory
    as well
  • Technology that enables verification enables
    exploratory creation of theories with explanatory
    power

12
What can we expect from an ABS?
  • To address validation, let us ask, what can we
    expect from a theoretically perfect ABS?
  • Even if the agent based model was completely
    correct, it still could not do long term
    prediction
  • The social world is full of Schelling Points
    arbitrary phenomena
  • We can expect it to display similar patterns to
    the real world, but not the exact data of the
    real world
  • It should have the same correlative patterns
  • Links between events in a simulation should have
    a similar strength to links between corresponding
    events in the real world
  • It should develop a distribution of plausible
    results similar to the real world
  • Tested by separating the test set from the
    training set
  • It should be able to make a short term prediction
    of types of phenomena
  • A live connection to data is essential
  • An agent based simulation is a theory
  • It is a theory represented in a form amenable to
    computation
  • The theory that best matches the (patterns in)
  • data is the best theory

13
Validating Agent-Based Simulations
  • Data-Based vs. Theory-Based Agent models How do
    we simulate both theory and data well?
  • The trajectory of a theory-based simulation can
    be made to pass through particular data
  • Random number massaging
  • Co-evolutionary seeding
  • Because the data emerges from the simulation
    itself, it models the next state better
  • It is validated if it models not only patterns in
    data, and the social literature well, but it also
    models causation well
  • Ockhams razor If many known phenomena emerge
    from a few known phenomena, you have modeled a
    cause well

14
Agent-Based Simulation and Accreditation
  • Rating for a usage in a domain is based on
    correctness of past usage in that domain
  • Pattern-based correctness
  • Social Science simulations are so complex, that
    scientific insight is needed in each new
    application
  • There is no way to generalize what tool will
    always be good in advance for what domain
  • Accreditation efforts should be devoted to
    confirming that a simulation does have expressive
    and explanatory power after the tool is chosen
    for the application
  • When is a model ready for use in analysis?
  • When it predicts patterns in data and the
    occurrence of types of events consistently when
    given new data

15
Myths of Agent-Based VVA
  • Chaos theory says there is no order, and any
    small change makes a big change in the outcome
  • The social world is full of order and homeostasis
  • The cause of emergent phenomena is so complex
    that it is unknowable
  • Cause is knowable because it is contained in the
    box
  • Scientific experiments can tease out cause
  • Computer experiments can hold all else the same
    better than real world experiments can
  • Statistics can find cause in Monte Carlo ABS

16
Implications for Existing VVA Techniques
  • Exploratory Space and Risk Analysis
  • Testing simulations at the boundaries where it
    matters
  • Nonlinearities in agent-based simulation means we
    dont know where it matters
  • Agency can be taken advantage of in strategic
    data farming
  • Bottom-up VVA
  • Making sure that the lower level is VVAd and
    that will take care of the upper level
  • But you dont know what to emphasize in the lower
    level until after the emergence happens

17
Summary
  • Agent Based Simulations model Agency
  • ABS are best used when mental processes and
    dynamic networks are important
  • ABS may be typed according to two dimensions
  • Cognitive/Reactive
  • Data-Based/Theory-Based
  • There is hope for Agent Based Simulation
    Verification, Validation and Accreditation
  • We have ways to measure
  • Similar patterns to the real world correlative
    data
  • Match to the social theory in literature
  • Explanatory power (Ockham's Razor)
About PowerShow.com