Navigating the SPS Toolbox: From Qualitative to Quantitative and Points InBetween

1
Navigating the SPS Toolbox From Qualitative to
Quantitative and Points In-Between

• Greg Paoli
• Decisionalysis Risk Consultants, Inc.
• Ottawa, Canada

2
Roadmap

• Context
• Risk Assessment vs. Scientific Characterisation
• Dimensions of Validity in Risk Assessment
• The Qual-Quant Continuum
• Methodological Research
• Final Thoughts

3
Personal Context

• Multi-disciplinary, multi-risk experience
• Public safety inspection planning, climate
  change, conservation of museum antiquities,
  product liability and litigation risk, medical
  devices, air quality advisories, nutritional risk
  in aboriginal communities, drinking water
  treatment, avalanche risk.
• Microbial food safety risk assessment
• FAO/WHO expert consultations on risk assessment
• FAO/WHO guidance development panel

4
Problem Context

• FAO/WHO risk characterisation guidance
  development to be considered by the Codex
  Alimentarius Commission
• Need to address qualitative risk assessment
• Recognition of the burden of quantitative risk
  assessment
• Sensitive to resource constraints
• Foodborne Microbial Risk Assessment is dominated
  by quantitative analysis
• Few examples and few proponents of qualitative
  risk analysis in food safety

5
Risk Assessment Essentials

• Evidence is processed in order to generate
  statements of probability of individual events
  which are combined to determine the probability
  of an adverse outcome of interest.
• The primary value-added feature is in the
  inference of the probability of adverse outcomes
  by appropriately combining a formal
  representation of the risk generating system with
  the rules of inferring probability.

6
Risk Assessment Essentials

• Appropriate characterization of the system for
  the purpose of inferring probability will
  generally require representing and incorporating
  variability in various phenomena since this is
  known to be critical in estimating the
  probability of adverse outcomes.

7
Risk Assessment and Scientific Characterisation

• At what point does an argument based on
  scientific evidence become a risk assessment?
• Documentation headings?
• Conformance with a bubbles-and-arrows framework?
• Discussion of Probability?
• Inference of Probability
• Is it useful to make a distinction?

8
Dimensions of Validity in Risk Assessment
9
Dimensions of Validity

• Problems with Empty Terminology
• Sound science
• Transparency
• Rational, Logical
• Consider likelihood and consequences
• Qualitative and Quantitative are Equally Valid
• Appropriate use of Bubbles and Arrows

10
Dimensions of Validity

• Validity is
• Decision-context dependent
• Multi-dimensional
• Measured on a continuum
• There is no consensus on which dimensions carry
  the most weight
• Tradeoffs are inevitable

11
Evidentiary Dimensions of Validity

• Quality of Evidence
• Transparency of Evidence
• Quality of Inference
• Transparency of Inference
• Strict or Real Transparency?
• Appropriateness of Outcome Measure

12
Managerial Dimensions of Validity

• Timeliness
• Resource Requirements
• Consistency
• Meta-Dimensions
• Inclusiveness (Analytic-Deliberative)
• Comfort in Subservient Roles

13
The Qual-Quant Continuum

• Narrative
• Narrative with Risky Terminology
• Structured Qualitative
• Semi-Quantitative
• Labels and Combinatorial Rules
• Scoring Systems
• Quantitative
• Deterministic
• Probabilistic

14
The Qual-Quant Continuum

• Royal Decree
• Narrative with Risky Terminology
• Structured Qualitative
• Semi-Quantitative
• Labels and Combinatorial Rules
• Scoring Systems
• Quantitative
• Deterministic
• Probabilistic
• Supercalifragilistic (SEAD)

15
Quality and Transparency of Evidence
Narr
Qual
Prob
Ferrari
Semi-Quant
Determ
16
Quality of Inference
Quality of Inference
Narr
Qual
Prob
Ferrari
Semi-Quant
Determ
17
Transparency of Inference
Strict
Real
Narr
Qual
Prob
Ferrari
Semi-Quant
Determ
18
Timeliness
Narr
Qual
Prob
Ferrari
Semi-Quant
Determ
19
Resource Requirements
Narr
Qual
Prob
Ferrari
Semi-Quant
Determ
20
Which Artist is Correct?

• You cant always get what you want,but if you
  try sometime, youll find you get what you
  need
• You Cant Get What You Want, Til You Know What
  You Want

21
Methodological Research is Critically Needed

• Robustness of Inference in Qualitative and
  Semi-Quantitative Risk Assessment
• Characterisation of the Limits of Simplifications
• Rapid Risk Assessment Development
• Produce, Test, Revise, Test, Watch
• Ban Extra-Label Use
• Innovation to alleviate tension

22
Final Thoughts

• At what point does a scientific characterisation
  become a risk assessment?
• Which dimensions of validity do we really value?
• Why, why, 1000 times why?
• Reconsider the double standard
• Qualitative risk assessment is inherently high
  risk

23
Thank you for your attention!

• Greg Paoli
• Decisionalysis Risk Consultants, Inc.
• 1831 Yale Avenue
• Ottawa, ON
• gpaoli_at_AnalyzeRisk.com
• www.AnalyzeRisk.com