Validation and Verification of Simulation Models - PowerPoint PPT Presentation

Loading...

PPT – Validation and Verification of Simulation Models PowerPoint presentation | free to download - id: 61ac8-OWVkZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Validation and Verification of Simulation Models

Description:

Techniques for Verification of Simulation Models. Use good programming practice: ... We make n experiments (using historical data) and compute Xj - Yj (for j = 1, 2, ... – PowerPoint PPT presentation

Number of Views:190
Avg rating:3.0/5.0
Slides: 21
Provided by: Industrial82
Learn more at: http://www.cs.bc.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Validation and Verification of Simulation Models


1
Validation and Verification of Simulation Models
2
Outline
  • Introduction
  • Definitions of validation and verification
  • Techniques for verification of simulation models
  • Techniques for validation of simulation models
  • Statistical Methods for Comparing real-world
    observations with simulation output data
  • Inspection Approach
  • Confidence-Interval Approach
  • Summary

3
Introduction
  • One of the most difficult problems facing the
    simulation analyst is determining whether a
    simulation model is an accurate representation of
    the actual system being studied ( i.e.,
    whether the model is valid).
  • If the simulation model is not valid, then any
    conclusions derived from it is of virtually no
    value.
  • Validation and verification are two of the most
    important steps in any simulation project.

4
What are Validation and Verification?
  • Validation is the process of determining whether
    the conceptual model is an accurate
    representation of the actual system being
    analyzed. Validation deals with building the
    right model.
  • Verification is the process of determining
    whether a simulation computer program works as
    intended (i.e., debugging the computer program).
    Verification deals with building the model right.

Validation
Conceptual Model
Real -World System
Verification
Validation
Simulation Program
5
Techniques for Verification of Simulation Models
  • Use good programming practice
  • Write and debug the computer program in modules
    or subprograms.
  • In general, it is always better to start with a
    moderately detailed model, and later embellish,
    if needed.
  • Use structured walk-through
  • Have more than one person to read the
    computer program.
  • Use a trace
  • The analyst may use a trace to print out some
    intermediate results and compare them with hand
    calculations to see if the program is
    operating as intended.

6
Techniques for Verification of Simulation Models
  • Check simulation output for reasonableness
  • Run the simulation model for a variety of input
    scenarios and check to see if the output is
    reasonable.
  • In some instances, certain measures of
    performance can be computed exactly and used for
    comparison.
  • Animate
  • Using animation, the users see dynamic displays
    (moving pictures) of the simulated system.
  • Since the users are familiar with the real
    system, they can detect programming and
    conceptual errors.

7
Techniques for Verification of Simulation Models
  • Compare final simulation output with analytical
    results
  • May verify the simulation response by running a
    simplified version of the simulation program with
    a known analytical result. If the results of the
    simulation do not deviate significantly from the
    known mean response, the true distributions can
    then be used.
  • For example, for a queuing simulation model,
    queuing theory can be used to estimate steady
    state responses (e.g., mean time in queue,
    average utilization). These formulas, however,
    assume exponential interarrival and service times
    with n servers (M/M/n).

8
Techniques for Validation of Simulation Models
  • A three-step approach for developing a valid and
    credible model
  • 1. Develop a model with high face validity
  • The objective of this step is to develop a
    model that, on the surface, seems reasonable to
    people who are familiar with the system under
    study.
  • This step can be achieved through discussions
    with system experts, observing the system, or the
    use of intuition.
  • It is important for the modeler to interact
    with the client on a regular basis throughout
    the process.
  • It is important for the modeler to perform a
    structured walk-through of the conceptual
    model before key people to ensure the
    correctness of models
    assumptions .

9
Techniques for Validation of Simulation Models
  • 2. Test the assumptions of the model empirically
  • In this step, the assumptions made in the initial
    stages of model development are tested
    quantitatively. For example, if a theoretical
    distribution has been fitted to some observed
    data, graphical methods and goodness of fit tests
    are used to test the adequacy of the fit.
  • Sensitivity analysis can be used to determine if
    the output of the model significantly changes
    when an input distribution or when the value of
    an input variable is changed. If the output
    is sensitive to some aspect of the
    model, that aspect of the model must be modeled
    very carefully.

10
Techniques for Validation of Simulation Models
  • 3. Determine how representative the simulation
    output data are
  • The most definitive test of a models validity is
    determining how closely the simulation output
    resembles the output from the real system.
  • The Turing test can be used to compare the
    simulation output with the output from the real
    system. The output data from the simulation can
    be presented to people knowledgeable about the
    system in the same exact format as the system
    data. If the experts can differentiate between
    the simulation and the system outputs, their
    explanation of how they did that should improve
    the model.
  • Statistical methods are available for comparing
    the output from the simulation model with those
    from the real-world system .

11
Statistical Methods for Comparing Real-World
Observation With Simulation Output Data
  • Suppose are observations
    from a real-world system and
    are output data from
    the simulation model.
  • Both, the real-world system outputs and
    simulation outputs are almost always
    non-stationary (the distribution of the
    successive observations change over time) and
    autocorrelated(the observations are correlated
    with each other).
  • Therefore, because classical statistical tests (
    the t-test, chi-square test, K-S test, etc.)
    assume I.I.D data, they can not directly be used
    to compare the two data sets to determine model
    validity.

12
Statistical Methods for Comparing Real-World
Observation With Simulation Output Data
  • 2 approaches for comparing the outputs from the
    real-world system with the simulation outputs
    are
  • Inspection Approach
  • Confidence-Interval Approach

13
Inspection Approach
  • Run the simulation model with historical system
    input data (e.g., actual observed interarrival
    and service times) instead of sampling from the
    input probability distributions, and compare the
    system and model output data.
  • The system and the model experience exactly the
    same observations from the input random
    variables.
  • This approach results in model and system outputs
    being positively correlated.

14
Inspection Approach
Historical system input data
Historical system input data
Simulation model
Actual system
System output data
Model output data
15
Inspection Approach
  • If X is the output from the real-world system and
    Y is the corresponding output from the model, we
    are interested in estimating .
  • We make n experiments (using historical data) and
    compute Xj - Yj (for j 1, 2, …, n) as an
    estimate of . Note that Xj and Yj
    use exactly the same interarrival times and
    service times).
  • Xj and Yj can then be plotted such that the
    horizontal axis denotes time and the vertical
    axis denotes the real and simulated outputs. The
    user can then eyeball timepaths to see if the
    model accurately represents the real-world
    system.

16
Inspection Approach
  • Due to positive correlation between X and Y,
    Var (X-Y) is much smaller than if X and Y were
    independent which makes Xj - Yj a much better
    estimate of
  • Reason
  • -- In general,
  • Var (X-Y) Var (X) Var (Y) - 2Cov(X, Y)
  • -- If X and Y are independent, Cov (X, Y)0
  • and Var (X-Y) Var (X) Var (Y)
  • -- But, if X and Y are positively correlated,
    Cov(X, Y) gt 0 leading to a smaller value for Var
    (X-Y).

17
Inspection Approach
  • Inspection approach (also called trace driven
    method)may provide valuable insight into the
    adequacy of the simulation model for some
    simulation studies.
  • In fact, this may be the only feasible approach
    because of severe limitations on the amount of
    data available on the operation of some
    real-world systems (e.g., military situation).
  • Care must, however, be taken in interpreting the
    result of this approach.

18
Confidence-Interval Approach
  • A more reliable approach for comparing a
    simulation model with the real-world system.
  • Requires a large amount of data.
  • Suppose we collect m independent sets of data
    from the system and n independent sets of data
    from the model (m and n can be equal).
  • Let Xj be the average of observations in the jth
    set of system data with mean and let Yj be
    the output from the jth replication of the
    simulation model with .
  • The objective is to build a confidence- interval
    for

19
Confidence-Interval Approach
  • In the case of correlated outputs where Xj is
    correlated with Yj (e.g., using trace driven
    simulation), let m n and pair Xj s and Yjs.
    Let Zj Xj -Yj for j1, 2 …, n. Zjs are IID
    random variables with E(Zj) .
  • Let
  • and
  • Then, the approximate 100(1- ) percent C.I. is
  • If the confidence interval does not include a
    zero, then the observed difference between
    and is statistically different at level
    .

20
Summary
  • Validation and verification are two of the most
    important steps in any simulation study.
  • Validation is not something to be attempted after
    the simulation model has already been developed,
    and only if there is time and money still
    remaining. Instead, model development should be
    done throughout the entire simulation study.
About PowerShow.com