Loading...

PPT – Validation and Verification of Simulation Models PowerPoint presentation | free to download - id: 61ac8-OWVkZ

The Adobe Flash plugin is needed to view this content

Validation and Verification of Simulation Models

Outline

- Introduction
- Definitions of validation and verification
- Techniques for verification of simulation models
- Techniques for validation of simulation models
- Statistical Methods for Comparing real-world

observations with simulation output data - Inspection Approach
- Confidence-Interval Approach
- Summary

Introduction

- One of the most difficult problems facing the

simulation analyst is determining whether a

simulation model is an accurate representation of

the actual system being studied ( i.e.,

whether the model is valid). - If the simulation model is not valid, then any

conclusions derived from it is of virtually no

value. - Validation and verification are two of the most

important steps in any simulation project.

What are Validation and Verification?

- Validation is the process of determining whether

the conceptual model is an accurate

representation of the actual system being

analyzed. Validation deals with building the

right model. - Verification is the process of determining

whether a simulation computer program works as

intended (i.e., debugging the computer program).

Verification deals with building the model right.

Validation

Conceptual Model

Real -World System

Verification

Validation

Simulation Program

Techniques for Verification of Simulation Models

- Use good programming practice
- Write and debug the computer program in modules

or subprograms. - In general, it is always better to start with a

moderately detailed model, and later embellish,

if needed. - Use structured walk-through
- Have more than one person to read the

computer program. - Use a trace
- The analyst may use a trace to print out some

intermediate results and compare them with hand

calculations to see if the program is

operating as intended.

Techniques for Verification of Simulation Models

- Check simulation output for reasonableness
- Run the simulation model for a variety of input

scenarios and check to see if the output is

reasonable. - In some instances, certain measures of

performance can be computed exactly and used for

comparison. - Animate
- Using animation, the users see dynamic displays

(moving pictures) of the simulated system. - Since the users are familiar with the real

system, they can detect programming and

conceptual errors.

Techniques for Verification of Simulation Models

- Compare final simulation output with analytical

results - May verify the simulation response by running a

simplified version of the simulation program with

a known analytical result. If the results of the

simulation do not deviate significantly from the

known mean response, the true distributions can

then be used. - For example, for a queuing simulation model,

queuing theory can be used to estimate steady

state responses (e.g., mean time in queue,

average utilization). These formulas, however,

assume exponential interarrival and service times

with n servers (M/M/n).

Techniques for Validation of Simulation Models

- A three-step approach for developing a valid and

credible model - 1. Develop a model with high face validity
- The objective of this step is to develop a

model that, on the surface, seems reasonable to

people who are familiar with the system under

study. - This step can be achieved through discussions

with system experts, observing the system, or the

use of intuition. - It is important for the modeler to interact

with the client on a regular basis throughout

the process. - It is important for the modeler to perform a

structured walk-through of the conceptual

model before key people to ensure the

correctness of models

assumptions .

Techniques for Validation of Simulation Models

- 2. Test the assumptions of the model empirically
- In this step, the assumptions made in the initial

stages of model development are tested

quantitatively. For example, if a theoretical

distribution has been fitted to some observed

data, graphical methods and goodness of fit tests

are used to test the adequacy of the fit. - Sensitivity analysis can be used to determine if

the output of the model significantly changes

when an input distribution or when the value of

an input variable is changed. If the output

is sensitive to some aspect of the

model, that aspect of the model must be modeled

very carefully.

Techniques for Validation of Simulation Models

- 3. Determine how representative the simulation

output data are - The most definitive test of a models validity is

determining how closely the simulation output

resembles the output from the real system. - The Turing test can be used to compare the

simulation output with the output from the real

system. The output data from the simulation can

be presented to people knowledgeable about the

system in the same exact format as the system

data. If the experts can differentiate between

the simulation and the system outputs, their

explanation of how they did that should improve

the model. - Statistical methods are available for comparing

the output from the simulation model with those

from the real-world system .

Statistical Methods for Comparing Real-World

Observation With Simulation Output Data

- Suppose are observations

from a real-world system and

are output data from

the simulation model. - Both, the real-world system outputs and

simulation outputs are almost always

non-stationary (the distribution of the

successive observations change over time) and

autocorrelated(the observations are correlated

with each other). - Therefore, because classical statistical tests (

the t-test, chi-square test, K-S test, etc.)

assume I.I.D data, they can not directly be used

to compare the two data sets to determine model

validity.

Statistical Methods for Comparing Real-World

Observation With Simulation Output Data

- 2 approaches for comparing the outputs from the

real-world system with the simulation outputs

are - Inspection Approach
- Confidence-Interval Approach

Inspection Approach

- Run the simulation model with historical system

input data (e.g., actual observed interarrival

and service times) instead of sampling from the

input probability distributions, and compare the

system and model output data. - The system and the model experience exactly the

same observations from the input random

variables. - This approach results in model and system outputs

being positively correlated.

Inspection Approach

Historical system input data

Historical system input data

Simulation model

Actual system

System output data

Model output data

Inspection Approach

- If X is the output from the real-world system and

Y is the corresponding output from the model, we

are interested in estimating . - We make n experiments (using historical data) and

compute Xj - Yj (for j 1, 2, , n) as an

estimate of . Note that Xj and Yj

use exactly the same interarrival times and

service times). - Xj and Yj can then be plotted such that the

horizontal axis denotes time and the vertical

axis denotes the real and simulated outputs. The

user can then eyeball timepaths to see if the

model accurately represents the real-world

system.

Inspection Approach

- Due to positive correlation between X and Y,

Var (X-Y) is much smaller than if X and Y were

independent which makes Xj - Yj a much better

estimate of - Reason
- -- In general,
- Var (X-Y) Var (X) Var (Y) - 2Cov(X, Y)
- -- If X and Y are independent, Cov (X, Y)0
- and Var (X-Y) Var (X) Var (Y)
- -- But, if X and Y are positively correlated,

Cov(X, Y) gt 0 leading to a smaller value for Var

(X-Y).

Inspection Approach

- Inspection approach (also called trace driven

method)may provide valuable insight into the

adequacy of the simulation model for some

simulation studies. - In fact, this may be the only feasible approach

because of severe limitations on the amount of

data available on the operation of some

real-world systems (e.g., military situation). - Care must, however, be taken in interpreting the

result of this approach.

Confidence-Interval Approach

- A more reliable approach for comparing a

simulation model with the real-world system. - Requires a large amount of data.
- Suppose we collect m independent sets of data

from the system and n independent sets of data

from the model (m and n can be equal). - Let Xj be the average of observations in the jth

set of system data with mean and let Yj be

the output from the jth replication of the

simulation model with . - The objective is to build a confidence- interval

for

Confidence-Interval Approach

- In the case of correlated outputs where Xj is

correlated with Yj (e.g., using trace driven

simulation), let m n and pair Xj s and Yjs.

Let Zj Xj -Yj for j1, 2 , n. Zjs are IID

random variables with E(Zj) . - Let
- and
- Then, the approximate 100(1- ) percent C.I. is
- If the confidence interval does not include a

zero, then the observed difference between

and is statistically different at level

.

Summary

- Validation and verification are two of the most

important steps in any simulation study. - Validation is not something to be attempted after

the simulation model has already been developed,

and only if there is time and money still

remaining. Instead, model development should be

done throughout the entire simulation study.