Expert Judgment Techniques and Distribution Sensitivity Analyses in RMPS Methodology

1
Expert Judgment Techniques and Distribution
Sensitivity Analyses in RMPS Methodology
Ricardo Bolado Lavín Nuclear Safety Unit,
Institute for Energy, European Commission
DG-JRC http// www.energyrisks.jrc.nl http//www
.jrc.nl
2
RMPS methodology
3
Identification of relevant parameters

• Use of the Analytical Hierarchy Process (AHP). It
  allows to create a hierarchy of parameters in the
  case studied (rank parameters according to their
  importance)
• Building a hierarchy to decompose the problem
• Definition of the top goal (top level in the
  hierarchy)
• Create lower levels searching for factors that
  affect the upper level and are affected by others
  in the lower level
• Locate basic parameters in the lowest part of the
  hierarchy
• Use of pair wise comparisons
• For each element in each level build a pair wise
  comparison matrix to assess importance of
  elements in the level just below it.
• Normalized eigenvalues of the pair wise matrix
  comparison provide the ranking
• Index available to detect inconsistent
  assessments.

4
Identification of relevant parameters (2)
5
Quantification of uncertainties

• The method to be applied depends on the quantity
  of data available
• Many data available
• Classical estimation methods Chi-square,
  Kolmogorov
• Not many data available
• Bayesian estimation methods
• Scarce or almost no data
• Expert judgment methods

6
Distribution sensitivity analysis

• What happens with our output variable (e.g.
  probability of success) if the distributions of
  our inputs are changed?
• There are several reasons for the experts to
  provide more than one input vector distribution
• Experts may show reluctance to provide accurate
  answers/distributions (under confident experts).
• Experts may acknowledge that the actual
  distribution could be slightly different from the
  one provided by them.
• In some cases the experts consider that the
  distribution should be divided into different
  distributions (conditional distributions).
• Sometimes experts just disagree and provide
  different distributions
• A check of the influence of those different
  distributions may be worthy.

7
The trivial solution

• Let us assume that we want to estimate the
  influence of changing the distribution of one
  input parameter
• Take a sample of size n under the original
  distribution
• Propagate the uncertainty through the model
• Estimate the reliability under the original
  distribution
• Take a sample of size n under the alternative
  distribution
• Propagate the uncertainty through the model
• Estimate the reliability under the alternative
  distribution
• Compare both reliabilities.

8
The trivial solution (2)

• Let us assume that we want to estimate the
  influence of changing the distribution of k input
  parameters
• Repeat the process described in the first three
  points of last slide.
• Repeat the process described in steps 4 to 7 in
  last slide k times.
• Total number of computer code runs n kn
• Let us assume that we want to estimate the
  influence of changing simultaneously the
  distribution of 2 or more input parameters
• The problem becomes a combinatorial problem
  (forget the trivial method).

9
Available acceptable solutions

• Two methods proposed by Beckman and McKay (1987)
• The re-weighting method (based on the idea of
  importance sampling)
• Provides unbiased estimators for the mean of the
  output variable
• The rejection method (based on the idea of
  acceptance-rejection sampling)
• Estimates in an unbiased way the distribution
  function of the output variable (adequate for our
  objectives)

10
The problem

• Let us consider a k components vector of input
  parameters X.
• Let us assume that two different distribution
  functions f1(x) and f2(x) may be assigned to that
  input parameter vector.
• Let us also assume that the first one is the
  reference one and the second one is the one we
  are addressing for sensitivity.
• For simplicity, let us consider only one output
  variable Y.
• Let us call F1(y) and F2(y) to the distribution
  functions of Y when X follows the distributions
  f1(x) and f2(x) respectively.
• The challenge
• To use only one sample to estimate simultaneously
  properties of F1(y) and F2(y) in order to compare
  them.

11
The rejection method

• There are two conditions for the application of
  this method
• First, as in the previous method, the support of
  f2(x), R2, must be contained in the support of
  f1(x), R1 and
• Second, the quotient f2(x)/f1(x) must be bounded.
  Let us assume that the bound is M.

12
The rejection method (2) - Implementation

• Step 1? To get a sample of size n of the input
  random vector under the reference distribution
  X1, X2, , Xn. Their actual values will be x1,
  x2, , xn.
• Step 2 ? To run the code for those n sets of
  inputs in order to get the corresponding sample
  of the output variable Y1, Y2,, Yn. Their
  actual values will be y1, y2,, yn.
• Step 3? For each sample xi, take a sample of the
  uniform distribution Vi between 0 and Mf1(xi).
• Step 4? To retain in the sample the corresponding
  output value Yiyi if the realisation vi of Vi is
  less or equal to f2(xi), otherwise reject that
  value from the sample.
• Step 5? To consider the values of Y remaining in
  the sample as a random sample of Y under f2(x).
  That sample of size kn will be used to build up
  an empirical distribution function that estimates
  the actual distribution function of Y under
  f2(x). Each step in that empirical distribution
  function will have a height 1/k.

13
The rejection method (3) - Example
14
The rejection method (4) - Example
15
Problem Inefficiency

• The efficiency of this method is 1/M (probability
  of a random realization of X obtained under f1(x)
  being retained as a random realization under
  f2(x)).
• Intuitive justification the quotient between the
  hypervolume under Mf1(x) and the hypervolume
  under f2(x) is M, so that, in the average,
  according to the acceptance criterion, only a
  fraction 1/M of the observations obtained under
  f1(x) will remain as a sample under f2(x)

16
Proposed solution
17
Proposed solution (Extended rejection method)

• Applying the rejection method an infinite number
  of times and averaging converges to a solution
  (red stepwise line in previous slide) that uses
  all the information in an optimum way
• The distribution obtained is a discrete random
  variable with probability
• for each sample value Y(xi)
• The estimator for the mean
• under the new distribution is

18
Application of the method

• RP2_B case
• Success criterion Pressure comes below 40 bar
  before the end of the scenario
• Possible variations in input distributions
• Mean varies up to 25 of the standard deviation
• Standard deviation varies up to 10

19
Application of the method
20
Application of the method
21
Conclusions

• There are methods available for phenomena and
  parameter identification and screening
• There are methods available for uncertainty
  quantification
• There are methods available for assessing
  (performance) reliability
• There are methods for computing distribution
  sensitivity
• Are there systems available to apply these
  methods on them?
• Fortunately it seems so!!!