Verification, Validation and Accreditation of AgentBased Simulations - PowerPoint PPT Presentation


PPT – Verification, Validation and Accreditation of AgentBased Simulations PowerPoint presentation | free to download - id: 327b6-OGVlM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Verification, Validation and Accreditation of AgentBased Simulations


An ABS is a simulation in which entities have 'agency' ... Why some Computational Social Scientists prefer ABS ... we expect from a theoretically perfect ABS? ... – PowerPoint PPT presentation

Number of Views:75
Avg rating:3.0/5.0
Slides: 18
Provided by: pae79
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Verification, Validation and Accreditation of AgentBased Simulations

Verification, Validation and Accreditation of
Agent-Based Simulations
  • Deborah Duong

  • To introduce Agent-Based Simulation
  • To propose measures of effectiveness for
    Agent-Based Simulation

What is an Agent-Based Simulation?
  • Agent-Based Simulation (ABS) is broadly defined
  • An ABS is a simulation in which entities have
  • Agents can perceive and behave in their
    environment based on goals
  • Agent-Based Simulation is used for modeling
    living systems
  • Biological and social systems
  • Non-living systems are mindless, and therefore
    dont have agency
  • The concept of emergence is important
  • Agents behave according to one set of rules
  • New patterns emerge from individual behaviors
  • Emergence is micro-macro integration

How does Agent-Based Simulation Compare ?
  • Other methods that dont involve agency or minds
    are also used to describe living systems
  • Discrete Event Simulation
  • Events of a process are scheduled to occur at
    discrete points
  • System Dynamics Simulation
  • Looks at the flow of fluid levels over time
  • Time delays are important
  • Social Networks
  • Patterns in the arrangement of entities to each
    other are important
  • These methods are at their best when modeling
    non-mental phenomena
  • Ecology
  • Predator-Prey cycles
  • The Economy
  • Cycles not based on beliefs (like the stock
    market is)
  • Any time entities act similarly
  • Everybody eats!
  • Non-agent simulation methods model flows and
    arrangements of averaged entities
  • Their State does not change, because entities
    are not modeled explicitly
  • They are not networked
  • They are viewed from an external, etic

Why some Computational Social Scientists prefer
  • Their preference depends on their feelings on the
    importance of agency and minds
  • They may believe that other tools are not as rich
  • Other tools tend to make heroic assumptions
  • They often can not model the crux of the problem
  • They are more descriptive than causal
  • North and Macal
  • We believe that in the future virtually all
    computer simulations will be agent-based because
    of the naturalness of the agent representation
    and the close similarity of agent models to the
    predominant computational paradigm of
    object-oriented programming.

Cognitive vs. Reactive Agents
Data-Based vs. Theory-Based ABS
Agent Based Simulation and VVA
  • Verification
  • Determination of whether a simulation expresses a
    theory well
  • Validation
  • Determination of whether a simulation has
    fidelity with the real world
  • Accreditation
  • Determination that a simulation is useful for
    analysis of a particular domain
  • Verification, Validation and Accreditation of
    agent based models is problematic
  • VVA originated in physics models
  • The nature of social science has implications for
    agent based VVA

Agent-Based Simulation and Verification
  • The more a simulation has the power to express a
    theory, the more the simulation is verified
  • A System Dynamics model of a verbal theory
    wouldnt have a high degree of verification
    unless that theory was about time-delays
  • The referent of any mathematical or simulation
    model is a theory
  • In physics based models, verification is doable
  • In physics-based models, verification is mainly
    about bugs
  • Replication, or using a different method to
    simulate the same theory, can help debug agent
    based social models
  • In social-science based agent models,
    verification is the central issue
  • Verification is about technology to represent an
  • Newton had the technology of the calculus
  • The technology to simulate social theories is not
  • For example, a social theory about human learning
  • may need a computer that can match a human in
  • With knowledge of available tools and creativity,
  • Verification is just a matter of good
    (scientific) taste, for now

The Social Literature as the Referent
  • Fitting raw data is not enough for verification
  • Data can be over-fitted
  • One could simulate by never addressing cause,
    by only making correlated things appear magically
  • Since why is not modeled, the simulation is not
    generally applicable
  • If it wont model a new situation, it wont model
    itself well either
  • If there are no causes a level under the
    phenomena you model, you are only describing, not
  • You can not explore the new levers to change
    outcomes, other than the ones you put in the
    simulation to begin with
  • Data should be fitted through a theory of social
  • Thoughtful models in the social literature are
    preferred to models from other fields
  • Just because we have the tools to describe time
    delays, physical phenomena, and epidemiology
    doesnt relate them to social theory
  • Knowledge of all tools is needed to model the
    richness of the social world
  • Tested by surveying the relative frequency of
    issues in the social literature and comparing to
    the relative frequency of issues in an ABS

Agent-Based Simulation and Validation
  • The more explanatory power an agent-based
    simulation has, the more the simulation is
  • A simulation model should match the data in the
    world in the way that its theory matches it
  • Validation of agent based simulation is dependant
    on verification If an agent based simulation is
    not first verified, it will not be valid
  • Validation of agent based simulation is dependant
    on the explanatory power of its referent theory
    as well
  • Technology that enables verification enables
    exploratory creation of theories with explanatory

What can we expect from an ABS?
  • To address validation, let us ask, what can we
    expect from a theoretically perfect ABS?
  • Even if the agent based model was completely
    correct, it still could not do long term
  • The social world is full of Schelling Points
    arbitrary phenomena
  • We can expect it to display similar patterns to
    the real world, but not the exact data of the
    real world
  • It should have the same correlative patterns
  • Links between events in a simulation should have
    a similar strength to links between corresponding
    events in the real world
  • It should develop a distribution of plausible
    results similar to the real world
  • Tested by separating the test set from the
    training set
  • It should be able to make a short term prediction
    of types of phenomena
  • A live connection to data is essential
  • An agent based simulation is a theory
  • It is a theory represented in a form amenable to
  • The theory that best matches the (patterns in)
  • data is the best theory

Validating Agent-Based Simulations
  • Data-Based vs. Theory-Based Agent models How do
    we simulate both theory and data well?
  • The trajectory of a theory-based simulation can
    be made to pass through particular data
  • Random number massaging
  • Co-evolutionary seeding
  • Because the data emerges from the simulation
    itself, it models the next state better
  • It is validated if it models not only patterns in
    data, and the social literature well, but it also
    models causation well
  • Ockhams razor If many known phenomena emerge
    from a few known phenomena, you have modeled a
    cause well

Agent-Based Simulation and Accreditation
  • Rating for a usage in a domain is based on
    correctness of past usage in that domain
  • Pattern-based correctness
  • Social Science simulations are so complex, that
    scientific insight is needed in each new
  • There is no way to generalize what tool will
    always be good in advance for what domain
  • Accreditation efforts should be devoted to
    confirming that a simulation does have expressive
    and explanatory power after the tool is chosen
    for the application
  • When is a model ready for use in analysis?
  • When it predicts patterns in data and the
    occurrence of types of events consistently when
    given new data

Myths of Agent-Based VVA
  • Chaos theory says there is no order, and any
    small change makes a big change in the outcome
  • The social world is full of order and homeostasis
  • The cause of emergent phenomena is so complex
    that it is unknowable
  • Cause is knowable because it is contained in the
  • Scientific experiments can tease out cause
  • Computer experiments can hold all else the same
    better than real world experiments can
  • Statistics can find cause in Monte Carlo ABS

Implications for Existing VVA Techniques
  • Exploratory Space and Risk Analysis
  • Testing simulations at the boundaries where it
  • Nonlinearities in agent-based simulation means we
    dont know where it matters
  • Agency can be taken advantage of in strategic
    data farming
  • Bottom-up VVA
  • Making sure that the lower level is VVAd and
    that will take care of the upper level
  • But you dont know what to emphasize in the lower
    level until after the emergence happens

  • Agent Based Simulations model Agency
  • ABS are best used when mental processes and
    dynamic networks are important
  • ABS may be typed according to two dimensions
  • Cognitive/Reactive
  • Data-Based/Theory-Based
  • There is hope for Agent Based Simulation
    Verification, Validation and Accreditation
  • We have ways to measure
  • Similar patterns to the real world correlative
  • Match to the social theory in literature
  • Explanatory power (Ockham's Razor)