Measurements

1
Measurements Validation

• R G Stamp
• Head of Department of ATM Research
• National Air Traffic Services Ltd, UK

2
What is Validation?

• EATMS Validation Strategy (EVAS)
• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance

3
What is Validation?

• EATMS Validation Strategy (EVAS)
• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance

4
What is Validation?

• EATMS Validation Strategy (EVAS)
• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance

5
What is Validation?

• EATMS Validation Strategy (EVAS)
• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance

6
What is Validation?

• EATMS Validation Strategy (EVAS)
• The process through which a desired level of
  confidence in the ability of a deliverable to
  operate in a real-life environment may be
  demonstrated against a pre-defined level of
  functionality, operability and performance
• Validation builds confidence that the system is
  fit for purpose

7
How to Validate
8
How to Validate
Analytic
modelling
Fast-time
Field test
simulation
Shadow-mode
Small-scale
trial
real-time
simulation
Operational
trial
Operations
Large-scale
real-time
simulation
9
Where to Validate
Small-scale
real-time
simulation
10
Where to Validate

• NATS
• DLR
• CENA
• NLR
• EEC
• real-life environment

Small-scale
real-time
simulation
11
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

12
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

13
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

14
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

15
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

16
Validation Methodology

• Consistent Application of Validation Methodology
  Across All Sites

pre-defined level of functionality, operability
and performance
17
Measures of Merit

• MoMs Chosen to Test Hypotheses
• Controller Workload
• Airspace Capacity
• Quality of Service
• Usability
• Safety
• Measurements Taken
• e.g. Workload ISA, TLX, R/T, ..
• Objective Subjective Observed

18
Analysis

• Descriptive Statistics

19
Analysis

• Descriptive Statistics
• Inferential Statistics
• significant at 5 confidence level

20
Analysis

• Descriptive Statistics
• Inferential Statistics
• significant at 5 confidence level
• Backed up by Controller Subjective Comments
• I can still handle the aircraft while doing the
  crossword!

21
Analysis

• Descriptive Statistics
• Inferential Statistics
• significant at 5 confidence level
• Backed up by Controller Subjective Comments
• I can still handle the aircraft while doing the
  crossword!
• desired level of confidence

22
Effectiveness of PHARE Validation

• Core Validation Group with Representation from
  all Participating Sites
• Common Methodology Crucial
• Methodology Improved with Experience
• Validation Conducted by Individual Sites
• Lessons Re-learnt /or Lost
• Effectiveness Diminished
• Validation Should be Applied Throughout Project -
  Not Just at the End

23
Summary

• Validation Methodology Major Contributor to
  Credibility of PHARE
• Forms the Backbone for much of the Validation
  Activities within European ATM RD

24
Summary

• Validation Methodology Major Contributor to
  Credibility of PHARE
• Forms the Backbone for much of the Validation
  Activities within European ATM RD