Calibration and Validation

1
Calibration and Validation
Christopher R. Bennett Director - Data Collection
Ltd.
2
Reliability of Results Depend On

• How well data provided represent the real
  conditions being analysed as understood by the
  model
• How well the predictions of the model fit the
  real behaviour and the interactions between the
  various factors and conditions to which it is
  applied

3
Application of Model

• Input Data
• Must have a correct interpretation of the input
  data requirements
• Have a quality of input data appropriate for the
  desired reliability of results
• Calibration
• Adjust model parameters to enhance the accuracy
  of its representation of local conditions

4
Steps in Analysis
5
Calibration Focus

• Road User Effects
• The models must predict the correct magnitude of
  costs and relativity between components
• Road Deterioration and Works Effects
• The models must reflect local pavement
  deterioration rates and maintenance
  practices/effects

6
HDM Development

• Used structured mechanistic-empirical approach
• Pavement deterioration validated against four
  major overseas studies
• HDM applied in over 100 countries with varied
  climate and pavement types
• Found to give reasonable predictions when
  calibrated correctly

7
RUE Calibration - Canada
8
Calibration Levels

• Level 1 Basic Application
• Addresses most critical parameters
• Desk Study
• Level 2 Calibration
• Measures key parameters
• Conducts field surveys
• Level 3 Adaptation
• Major field surveys to requantify fundamental
  relationships

9
Hierarchy of Effort
10
Level 1 - Application

• Required for ALL HDM analyses
• Once off set-up investment for the model
• Primarily based on secondary sources
• Assumes bulk of HDM default values appropriate

11
Level 2 - Calibration

• Uses local measurements to verify and adjust
  predictions
• Requires more data collection and higher precision

12
Level 3 - Adaptation

• Comprised of
• Improved data collection
• Fundamental research
• Leads to more accurate data by observing over
  long time period
• Often leads to alternative local relationships

13
RUE Model Calibration Priorities
14
Full Details on RUE Calibration

• Specific details on how to calibrate the RUE
  model are given in the HDM-4 Calibration and
  Adaptation Guide as well as the book Modelling
  Road User and Environmental Effects in HDM-4.

15
Bituminous Pavement Deterioration Priorities
16
Full Details on PDWE Calibration

• Specific details on how to calibrate the PDWE
  model are given in the HDM-4 Calibration and
  Adaptation Guide

17
Reliability Concepts

• A model is representation of reality
• How well the model predictions reflect reality
  depends on
• the validity of the underlying relationships
• the accuracy and adequacy of the input data
• calibration factors used in the analysis

18
Bias and Precision

• Only way of assessing models reliability is by
  comparing its predictions to known data
• Need to take into account two considerations
• Bias
• Precision

19
Combinations of Bias and Precision
20
Correction Factors

• Used to correct for bias
• Two types of factors
• Rotation (CF Observed/Predicted)
• Translation (CF Observed - Predicted)
• Translation factors shift the predictions
  vertically rotation factors adjust the slope

21
Rotation and Translation Factors
22
Bias and Precision in Input Data
23
Important Considerations

• Must calibrate over full range of values likely
  to be encountered
• Must have sufficient data to detect the nature of
  bias and level of precision
• High correlation (r2) does not always mean high
  accuracy there can still be significant bias

24
RDWE Calibration - 1

• Simulation of Past
• take sample of roads with historical data
  (traffic, design, etc.)
• simulate deterioration from construction to
  current age
• compare results
• Average predicted condition should be similar to
  current condition

25
RDWE Calibration - 2

• Controlled Studies
• collects detailed data over time on traffic,
  roughness, deflections, condition, rut depths
• sections must be continually monitored
• long-term (5 yr) commitment to quality data
  collection

26
Road User Effects

• Some data available from field studies other
  from controlled experiments
• Can verify using tariff surveys

27
RUE - Fleet Surveys

• In many developed countries data are available
  from vehicle fleet management companies
• Depending on availability and level of
  disaggregation may offer scope for calibration of
  both level and relative contributions of RUE
  components

28
What to Focus On

• HDM-III had about 80 data items and model
  parameters HDM-4 over 100
• To assist users, conducted sensitivity tests and
  defined impact elasticities
• Grouped data into ranges

29
Sensitivity Classes
30
(No Transcript)
31
(No Transcript)
32
Importance of Data

• Accuracy of data has major impact on predictions

33
Information Quality Levels - IQL
34
IQL Levels

• IQL - 1 Fundamental Research
• many attributes measured/identified
• IQL - 2 Project Level
• detail typical for design
• IQL - 3 Programming Level
• few attributes, network level
• IQL - 4 Planning
• key management attributes
• IQL - 5 Key Performance Indicators

35
Description of IQL Levels
36
Converting Data Units

• Not necessary to collect data in same units as
  HDM
• Can develop transfer functions using parallel
  studies
• Functions may be equations or tables

37
Relating Local Data to HDM-4
38
Steps to Resolving Data Issues

• Establish IQL given the required decision level
  and the data collection resources
• Sort local data into format suitable for
  transformation
• Determine transformations between local data and
  HDM
• Apply transformation relationships to local data

39
Can We Believe the Output?

• Yes, if calibrated
• HDM has proved suitable in a range of countries
• As with any model, need to carefully scrutinise
  output against judgement
• If unexpected predictions problem with (a) data
  (b) calibration (c) the models, or (d) your
  judgement