Validation

1
Validation Measurement Methods for the PHARE
Demonstrations

• R A Whitaker
• Validation Project Leader

2
Analysis Is Not Easy!
3
Unreasonable Demands
4
Not Understood
5
Does Anyone Listen?
6
European Wide Validation

• NATS
• DLR
• CENA
• NLR
• EEC

7
European Wide Problems!
8
European Wide Solution
PHARE Validation Methodology
9
What is Validation?

• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance
• Validation builds confidence that the system is
  fit for purpose

10
Methodology Matureswith Time
PC/TC PD1 PD2 PD2
PD3 PD1
PD1
11
PD1 Validation Planning

• Experimental Design

12
PHARE Experimental Design

• Comparison To Baseline
• For Users to Understand Results - Baseline must
  be Close to Current System

13
PHARE Experimental Design

• Comparison To Baseline
• For Users to Understand Results - Baseline must
  be Close to Current System
• Progressive Introduction of Experimental
  Variables in New Organisations

14
PHARE Experimental Design

• Comparison To Baseline
• For Users to Understand Results - Baseline must
  be Close to Current System
• Progressive Introduction of Experimental
  Variables in New Organisations

15
PHARE Experimental Design

• Comparison To Baseline
• For Users to Understand Results - Baseline must
  be Close to Current System
• Progressive Introduction of Experimental
  Variables in New Organisations

16
PD1 Validation Planning

• Experimental Design
• Identifying Data Collection
• Objective Data
• What How Format
• Subjective Data
• Questionnaires
• Debriefs
• Concentration on Controller Workload
• Instantaneous Self Assessment (ISA)
• NASA TLX (paper)

17
PD1 Validation Execution

• Data Collected In One File
• Tedious data analysis
• No Recorded Observations
• so-and-so said such-and-suchI think
• Limited Material For Debrief Discussions
• Video Data - but limited Controller Cognitive
  De-Brief

18
PD1 Validation Execution

• No Real Thoughts on Analysis Techniques
• Descriptive Statistics
• Appropriate for Nominal Data?
• Inferential Statistics Employed as Analysis
  Progressed
• Allowed Confidence Intervals
• Limited Considerations to Measures of Merit
• Capacity fn(workload)?
• Quality of Service

19
PC/TC, PD1, PD2, PD2 Validation Preparation

• Clearly Defined Aims
• Objectives
• Hypotheses
• Measures of Merit
• Controller Workload
• Quality of Service
• Capacity
• Usability
• Data Required
• Accessible Storage
• Formal Observation
• Psychologists

20
Formal Documentation

• Engineering Plan
• Layout, tools, equipment
• Analysis Plan
• Aims, objectives, H0, H1
• Design
• Scenario,Traffic samples
• No. exercises, Duration
• Data recording
• Organisations
• Ensures matched pairs
• Measurements
• Analysis technique, Stat tests

PHARE
21
PC/TC, PD1, PD2, PD2 Validation Execution

• Use of both Descriptive and Inferential
  Statistics
• Defined by Analysis Plan
• Supported with Subjective And Formally Observed
  Data
• Immediate Input to Debrief
• Electronic TLX
• Trajectory plots
• Relatively Small and Well Informed Core Group of
  validators

22
PD3 Validation Planning

• Validation Group expanded to non-analysts
• Needed to Disseminate Knowledge - Documented
• Experimental Plan Requirements
• Analysis Plan Requirements
• Aims, Objectives, H0, H1
• .
• Analytical Methods
• Acted as Consultancy

23
PD3 Validation Execution

• Planning Worked Well - Execution of Analysis
  Limited to CENA
• Documentation can Only be a Guide to
  Practitioners of Best Practice
• Cant Learn Analysis / Validation from a Book

24
PD1 Validation Planning

• Planning - routine
• Fast-Time Simulation (TAAM) Introduced
• Development of airspace, traffic samples
• Early examination of potential problem areas
• Training
• Introduced Temporary Operating Instructions

25
PD1 Validation Execution

• Caught out by Complacency
• Capacity measures used to date only applicable to
  non-changing airspace
• Cannot compare capacity between different
  airspace designs as used in PD1
• Initiated research study

26
Effectiveness of PHARE Validation

• Core Validation Group with Representation from
  all Participating Sites
• ...those involved in the validation should have a
  thorough understanding of the operational concept
  under investigation and of any restrictions/specif
  ics of the simulation environment...
• Common Methodology Crucial
• traceability of all input and output data is
  important for correct analysis and can not be
  over-emphasised...
• ...the metrics applied in a research programme
  like PHARE must be relevant to the performance of
  real life ATM...

27
Effectiveness of PHARE Validation

• Methodology Improved with Experience
• the Validation Methodology avoided disruptive
  data collection. The use of intrusive
  measurements like eye tracking and heart-rate
  variability measurements could be valuable
  additions...
• the application of the appropriate technology to
  perform the investigations is very important. The
  early choice of PHARE to perform real time
  simulations has put a too strong focus on this
  type...

28
Effectiveness of PHARE Validation

• Validation Should be Applied Throughout Project -
  Not Just at the End
• ...there has been too little input from the
  Validation project in the beginning of the PD/3
  project. In hindsight it can be said that a
  stronger focus on the validation objectives of
  the project should have led to a better project
  definition...

29
Validation is a Living Subject

• PHARE Validation Methodology developed over time
  and through experience
• Methods are documented

30
Validation is a Living Subject
31
Validation Measurement Methods for the PHARE
Demonstrations

• R A Whitaker
• Validation Project Leader
• next