Experts

1
Experts consensus building on technology
risks(Expert judgments on phytoremediation The
role of self-confidence in averaging procedures
and Formative Consensus Building (FCP) for
predicting technology risks submitted)

• Roland W. Scholz

2
What will be told
Overview

• Theoretical motivation Statistical versus
  consensus building procedures the role of
  expert confidence
• The case of phytoremediation in Dornach
• The situation
• Technology application, uncertainties and
  technology performance
• 3. The procedure
• 4. Results
• Quantitative results Is averaging about all
  experts the best strategy?
• Qualitative results What are the potential and
  limits of consensus building procedures
• 5. Conclusions/Discussion

3
What to do, if
1. Motivation

• you have a new technology/medicine/educational
  program at hand and want to reduce the number of
  accidents/diseases/failure rate
• the situation of technology application is
  overly complex (cause impact relationships are
  multi-layered) and not completely known
• empirical evidence is limited (no von
  Mises-Reichenbach situation)
• Expert opinions diverge

4
and you are interested in
1. Motivation

• the range of outcomes after applying Tech A (i.e.
  statements such as The failure/ mortality rate
  will be between x and y.)
• the probability distribution on the reduction
  rate r p(r lt C) z

5
Two approaches
1. Motivation

• A) Statistical models (e.g. Johnson, Budescu,
  Wallsten, 2001 Averaging when maximizing
  independence among experts in formative
  measurement procedures)
• B) Consensus building procedures (e.g. Susskind,
  1999 Organizing open but mediated processes on
  what will be judged and which questions to be
  answered)

6
The situation in DornachLarge scale
contaminantion with Cadmium, Copper, and Zinc
2. The case
7
2. The case
8
Is the flowerful technology of phytoremediation
also a powerful one?
2. The case
9
2. The case
How Phytoremediation Works
Cropping conditions
Acquisition
Soil parameters pH, clay, conductivity,
Zn
Cu
10
Key questions Range/Bounds
Dependent variables

• If a model lot will be treated for ten years,
  the cadmium (copper, zinc) concentration will
  have a value between ___ mg Cd/Cu/Zn per kg dry
  matter of soil and ___ mg Cd/Cu/Zn per kg dry
  matter of soil?
• Expert ks estimation of lower bound
  concentration of the pollutants Cd/Cu/Zn
• Expert ks estimation of upper bound
  concentration of pollutants Cd/Cu/Zn

11
Key questions Probabilities on Remaining
Concentrations
Dependent variables

• Experts probability judgments on attaining a
  remaining degree of contamination r
• Question on p(remaining concentrationltr)
• Cd 80, 20, 50, 91, 90, 99, 30, 70, 1,
  40, 60, 10
• Cu 90, 99, 30, 70, 1, 40, 60, 10
• Zn 80, 30, 10, 99, 50, 95, 20, 1

12
Fishing in a pool of experts
Expert sample

• Large scale eight-year national environmental
  research program on soil remediation
• Project cluster of six projects on
  phytoremediation in Dornach (about 25
  researchers)
• 10 Experts from this cluster with backgrounds
  biology, chemistry, environmental engineering,
  mathematics, decision sciences and specialized
  knowledge on soil chemistry, biological
  mechanisms of heavy metal accumulation in plants,
  sampling and data analysis, or designing
  large-scale remediation engineering applications

13
Procedure

• Experts got detailed (anonymized) information
  about all experts judgments
• Consensus building workshop
• Signing a public statement

• What could/should be answered (sample lot, soil
  parameters, technology, key questions)
• Gathering and disseminating documented expertise
  (Multi-disciplinary state of the art knowledge
  79 pages)
• Questionnaire with key questions on
• Ranges
• Probabilities of attaining certain reductions
  (ca. 10 reduction rates asked per heavy metal)

14
H1 Experts confidence provides validity
4. Hypotheses

• Experts that feel more confident are more valid
  in the sense that they deviate less from the
  real/superexperts judgments
• Further The judgments of the high confidence
  group is more homogeneous than a low confidence
  group

15
H2 Statistical models to be compared
4. Hypotheses

• High confidence average among high confidence
  experts (N 4)
• Low confidence Average among low confidence
  experts (N 5)
• Average all (N 9)
• Median (N 9)
• Maxcorr Average among high correlated experts (N
  4)
• Mincorr Average among low correlated expert

16
H2 Trucating provides higher validity
4. Hypotheses

• Averaging only the medium responses (only the
  judgments of the inner 50 truncated
  distribution) improves validity The median
  expert does fine

17
H3 Showing low correlations in an expert pool is
not an indicator of expertise
4. Hypotheses

• Higher correlated experts provide more valid mean
  estimates (compared to a superexpert) than low
  correlated experts
• (In contradiction to Johnson et al. 2001)

18
H4 Consensus Building does/ does not provide
new results
4. Hypotheses

• Not a straight hypothesis more an exploratory
  one
• Consensus building provides more reliable/valid
  vs. fuzzier statements than statistical models
• The high confidence group is the base line

19
H1 Mean bounds of high and low confidence group
differ
4. Results

• Estimates of upper and lower bounds
• Means differ (Factor 2 in general not
  significant)
• Variances differ significantly low confidence
  experts are less homogeneous (show more variance)
• (see Table 1)

20
H1 Mean bounds of remaining concentr. of high and
low confidence group differ
4. Results
21
H1 Probability judgments of high and low
confidence group differ
4. Results

• Probability judgment on remaining concentrations
• High and low confidence group differ (rep. meas.
  ANOVA)
• Cd p lt .21 however interaction Probability x
  Confidence p lt .04
• Cu p lt .04
• Zn p lt .02

22
H1 Probability judgments of high and low
confidence group differ
4. Results
23
H2 High confidence experts are more valid
4. Results

• Estimates of upper and lower bounds
• High confidence experts show lower difference to
  a superexpert/real measurements in all 6
  estimates (Factor 2 however not significant)

24
H3 Self confidence provides validity
4. Results

• Mean sum of differences (absolute values) of
  experts and superexperts/real meas. probability
  judgments for different heavy metals

25
H3 The greenhorns are the greensLess confident
are more optimistic
4. Results
26
4. Results
Y-axis Deviations of probability judgments (sum
score) to a superexpert/meas. The Median is the
best high confidence does fine
Low conf (8)
High conf (2)
Truncin (3)
Truncout (6)
Model (rank)
Mincorr (8)
Maxcorr (3)
Median (1)
Average all (3)
27
H4 Qualitative statements consented

1. We all agree that the remaining concentration
   will be in the range between x and y (grey
   area) with a certain probability
2. For Cadmium The reduction will exceed 15 with
   low probability
3. For Zinc The Majority believes that the
   remaining concentration will be between 93 and
   98

28
Conclusions

• The Formative Consensus Building method (i.e., a
  structured, formative, anonymous method
  organized by an independent facilitator) should
  include
• Cooperative definition of the judgmental task
• A common knowledge base
• Statistical procedures of integrating judgments
  (better than fuzzy workshop statements)
• The validation by a data based super-expert
  judgment is a good/ideal research strategy
• Measuring distributional knowledge is possible
  Statistical procedures do better than discursive
  ones take the median expert!
• High confidence experts and high correlated
  experts provide better judgments (if ....)