Statistical Design of Demining Experiments and Analysis by Logistic Regression - PowerPoint PPT Presentation

1 / 67
About This Presentation
Title:

Statistical Design of Demining Experiments and Analysis by Logistic Regression

Description:

Factor settings for the experiment Oberjettenberg (May 2003) ... factors are not (or almost not) confounded so that they can be estimated ... – PowerPoint PPT presentation

Number of Views:36
Avg rating:3.0/5.0
Slides: 68
Provided by: wilr5
Category:

less

Transcript and Presenter's Notes

Title: Statistical Design of Demining Experiments and Analysis by Logistic Regression


1
Statistical Design of Demining ExperimentsandAna
lysis by Logistic Regression
  • Peter-Th. Wilrich
  • Free University of Berlin
  • Institute for Statistics and Econometrics
  • ITEP Workshop
  • Reliability Tests for Demining
  • December, 16-17, 2003
  • Federal Institute for Materials Research and
    Testing (BAM)

2
  • Aim of the investigation
  • Response variables and influence variables
  • Factor settings for the experiment Oberjettenberg
    (May 2003)
  • Experimental design of the experiment
    Oberjettenberg (May 2003)
  • The experiments Benkovac (July 2003) and
    Oberjettenberg (November 2003)
  • The data for analysis
  • The model for analysis
  • Some results of the analysis
  • Conclusions and recommendations - concerning the
    results to be obtained in a future
    experiment- concerning the uncertainty of the
    results- concerning the experimental effort

3
1. Aim of the investigation
  • Primarily, the aim of each of the experiments was
    to compare different types of devices for mine
    detection with respect to their ability to detect
    mines.
  • For this reason, two performance measures, the
    detection rate and the false alarm rate, are
    estimated statistically on the basis of the data
    obtained in the experiments.
  • The detection rate DR is defined as the
    probability of detecting a mine.
  • The false alarm rate FAR is defined as the
    average number of false alarms per square meter.
  • Both rates do not only depend on the type of
    device, but on many other factors, for instance
    the soil, the operator, the type of mine, the
    depth of the mine,
  • The dependency of these two rates, DR and FAR, on
    the factors is to be estimated statistically, and
    the uncertainty of the estimates has to be
    reported.

4
2. Response variables and influence variables
5
2. Response variables and influence variables
6
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • An influence variable which is included in the
    experiment by setting or controlling its levels,
    is called a factor.
  • For many factors the designer of the experiments
    has to accept the levels being used in the
    experiment, for others he has the freedom to
    decide.

7
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • The following factors have been included into the
    experiment Oberjettenberg (May 2003)
  • In Oberjettenberg, the following test lanes have
    been available
  • a lane with a slightly uncooperative soil
  • three lanes with cooperative soil
  • They define a qualitative factor, called lane,
    i.e. it is measured on a nominal scale, with the
    levels lane 1, , lane 4, which represent the
    four soil types available.
  • It is treated as a systematic factor, i.e. we are
    interested in an estimation of the effect of each
    of the four soil types separately.

8
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • 8 operators had been available, some of them
    more, some less trained.
  • They define a qualitative factor, called
    operator, with the levels operator A, ,
    operator H, which represent the influence of
    the operator on the response variable.
  • It is treated as a random factor, i.e. we are
    not interested in an estimation of the effect of
    these 8 operators separately. Instead we look at
    these operators as being a sample out of a
    population of operators, and we are interested
    in the standard deviation as a measure of the
    variation of the influence of the operators on
    the response variable.

9
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • Type of mine and depth of mine
  • Unfortunately, the test lanes had to be accepted
    as they have been, i.e. equipped with about 30
    mines and other ITOPs in various depths in each
    lane.
  • Hence, it was not possible to set the types of
    mines and the depths as levels of factors.
    However, since the information was available,
    these influence variables have been included in
    the analysis (with difficulties in the
    interpretation of the results).

10
  • depth cm 1 2 3 4 5 6 7 8 10 12 15 16 17 20
    22
  • 100Cr6 - 16mm 0 0 0 0 0 0 0 0 0 0 0 0 0 2
    1
  • Maus 4 6 2 1 3 0 0 0 0 0 0 0 0 0
    0
  • O0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
    0
  • PMN 0 3 3 0 2 0 0 0 0 0 3 0 1 0
    0
  • PPM-2 0 5 4 2 1 0 0 0 0 0 0 0 0 0
    0
  • PT-Mi-Ba-III 0 0 0 0 2 1 1 2 0 0 0 0 0 0
    0
  • SchAMi DM 31 0 0 0 0 2 1 1 0 0 0 0 0 0 0
    0
  • TM-46 0 0 0 0 2 1 0 1 0 0 0 0 0 0
    0
  • TM-62 M 0 0 0 0 1 2 1 0 0 0 2 1 1 0
    0
  • TM-62 P2 0 0 0 0 2 1 0 0 0 0 0 0 0 0
    0
  • TM-62 P3 0 0 0 0 3 2 2 0 0 0 0 0 0 0
    0
  • large G0 0 0 0 0 0 0 0 0 3 1 0 0 0 0
    0
  • large I0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
    0
  • large K0 0 0 0 0 0 0 0 0 2 1 0 0 0 0
    0
  • small C0 0 0 0 0 3 0 1 0 0 0 0 0 0 0
    0
  • small E0 0 0 0 0 0 0 0 0 2 0 0 0 0 0
    0
  • small G0 0 0 0 0 2 0 1 0 0 0 0 0 0 0
    0

11
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • Type of the mine detection device
  • It was decided to include 4 mine detection
    devices into the experiment.
  • They define a qualitative factor, called
    devicetype, with levels devicetype 1, ,
    devicetype 4.
  • It is treated as a systematic factor since we
    are interested to estimate the effect of each of
    the device types on the response variable is our
    primary interest.

12
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • Specimen of the mine detection device
  • For each of the 4 mine detection devices two
    specimens have been asked for, in order to get to
    know the variation between specimens of one and
    the same device type, expressed as its component
    of standard deviation.
  • This is a qualitative factor, called specimen,
    with two levels specimen 1 and specimen 2.
  • It is treated as a random factor, nested in the
    factor devicetype, because the level specimen
    1 or specimen 2 is meaningful only for a
    particular type of device but not across the
    factor devicetype.

13
3. Factor settings for the experiment
Oberjettenberg (May 2003)
  • Sensibility of the device
  • It was decided to use each specimen of each type
    of device on a high level and on a low level of
    sensibility.
  • This defines a qualitative factor, called
    sensibility, with the two levels low and
    high.
  • It is treated as a systematic factor.
  • This concludes the list of factors having been
    investigated in the Oberjettenberg (May 2003)
    experiment.

14
4. Experimental design for the experiment
Oberjettenberg (May 2003)
  • A full factorial design consists of obtaining at
    least one value of the response variable for each
    factor level combination.
  • There are 4 x 8 x 4 x 2 x 2 512 factor level
    combinations.
  • Without any repetitions, this would mean 512
    runs of operators at these lanes, or 1024 with
    one repetition.
  • Such a large experiment did not fit into the
    budget and time frame and it is not necessary
    to estimate the two rates sufficiently accurate.
  • Hence, a suitable number of runs has to be
    selected out of these 512 factor level
    combinations.

15
4. Experimental design for the experiment
Oberjettenberg (May 2003)
  • A basic requirement is that the main effects of
    all factors shall be estimated uncorrelated with
    the estimates of other main effects.
  • Remembering the table of types of mines and depth
    of mines of the test fields in Oberjettenberg, it
    is immediately clear that this is not the case
    for these two influence variables. Some mine
    types are in these test fields only at one (or
    two different) depth, and on the other hand
    some depths are only connected with one (or two)
    mine types. If, with this experimental design as
    the basis, the estimated detection rate for a
    particular depth is reported, one could it as
    well interpret as the detection rate for a
    particular type of mine.
  • An uncorrelated estimation of all main effects
    (but not of all interaction effects) can be
    achieved if an orthogonal design is used. Here, a
    design based on a Graeco Latin square design, was
    used.

16
4. Experimental design for the experiment
Oberjettenberg (May 2003)
  • Example Day 1
  • Deminers A, B, C, D, E, F, G, H
  • Devices a, a, b, b, g, g, d, d

17
5. The experiments Benkovac and Oberjettenberg
(November 2003)
  • The experimental designs of the experiments
    Benkovac and Oberjettenberg (November 2003) are
    similar to the experimental design of
    Oberjettenberg (May 2003).
  • In the Benkovac experiment types of mines (and
    other targets) and depths of mines were included
    as factors with levels set so that the effects of
    these two factors are not (or almost not)
    confounded so that they can be estimated
    uncorrelated with each other.

18
  • Benkovac (July 2003)
  • depthcm 0 5 10 13 20
  • E0 0 4 0 0 0
  • G0 0 4 0 0 0
  • K0 0 4 0 0 0
  • PMA-1A 4 12 4 4 4
  • PMA-2 4 12 4 4 4
  • PMA-3 4 12 4 4 4
  • PROM-1 8 12 0 0 0
  • TMA-3 0 0 1 0 0
  • TMA-4 0 0 3 0 0
  • TMM-1 0 0 1 0 0
  • TMRP-6 0 0 3 0 0
  • ball 0 0 4 0 0

19
5. The experiments Benkovac and Oberjettenberg
(November 2003)
  • The Oberjettenberg (November 2003) actually
    consists of two experiments
  • one is a replication of the Oberjettenberg (May
    2003) experiment with some slight changes
    concerning the operators and the sensibility
    level.
  • For the other one the test lanes have been newly
    equipped with mines (and other targets) as
    follows

20
  • Oberjettenberg (November 2003)
  • depthcm 0 5 10 13 20
  • ball 0 0 3 0 0
  • C0 0 3 0 0 0
  • E0 0 3 0 0 0
  • G0 0 3 0 0 0
  • I0 0 3 0 0 0
  • K0 0 3 0 0 0
  • PMA-S 3 9 3 3 3
  • Maus 3 9 3 3 3
  • PMNMS3 3 9 3 3 3
  • TM-62 M 0 0 3 0 0

21
6. The data for analysis
  • The first data frame, to be used for the
    estimation of the detection rate, OJT1, has 6720
    rows (one for each situation in which a mine can
    be detected or not) and 11 columns (one for each
    variable). The following table shows the first 15
    and the last 15 rows of this data frame.

22
(No Transcript)
23
6. The Data for Analysis
  • The second data frame, to be used for the
    estimation of the false alarm rate, OJF1, has 256
    rows (one for each start of an operator) and 8
    columns (one for each variable). The following
    table shows the first 15 and the last 15 rows of
    this data frame.

24
(No Transcript)
25
7. The model for analysis
26
7. The model for analysis
27
7. The model for analysis
28
7. The model for analysis
29
7. The model for analysis
30
7. The model for analysis
31
7. The model for analysis
  • The new model is the special case of logistic
    regression in the class of generalized linear
    models introduced by Nelder and Wedderburn in
    1972 and treated in a monograph by McCullagh and
    Nelder in 1983.
  • We overcome all difficulties with the model
    assumptions and we solve problem 2., i.e. we can
    include mine type and mine depth into our
    determination of the detection rate, even in our
    case where these two influence variables have not
    been taken into account as factors.
  • However, we cannot any more estimate the
    parameters by the method of least squares.
    Instead, we have to use the method of Maximum
    Likelihood and it would not be possible to find
    a solution without a computer.

32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
7. The model for analysis
37
8. Some results of the analysis
  • The analysis with the generalized linear models
    was carried out for the four experiments
  • Oberjettenberg (May 2003)
  • Benkovac (July 2003)
  • Oberjettenberg (November 2003), part 1
  • Oberjettenberg (November 2003), part2

38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
  • Analysis of Deviance Table
  • Binomial model
  • Response detected
  • Terms added sequentially (first to last)
  • Df Deviance Resid. Df Resid.
    Dev
  • NULL 6719
    8640.743
  • depth 1 483.8379 6718
    8156.905
  • devicetype 3 32.7918 6715
    8124.114
  • lane 3 2.7913 6712
    8121.322
  • operator 7 70.8227 6705
    8050.500
  • depthdevicetype 3 2.5687 6702
    8047.931
  • depthlane 3 6.3224 6699
    8041.609
  • depthoperator 7 7.7671 6692
    8033.841
  • devicetypelane 8 2.6677 6684
    8031.174
  • devicetypeoperator 18 11.4115 6666
    8019.762
  • laneoperator 12 124.7781 6654
    7894.984

59
  • Analysis of Deviance Table
  • Initial Model
  • detected (depth devicetype lane
    operator)2
  • Final Model
  • detected depth lane operator
    laneoperator
  • Step Df Deviance Resid. Df
    Resid. Dev AIC
  • 1 6654
    7894.984 8026.984
  • 2 - devicetypeoperator 12 4.731628 6666
    7899.716 8007.716
  • 3 - devicetypelane 6 1.399623 6672
    7901.115 7997.115
  • 4 - depthoperator 7 9.655222 6679
    7910.771 7992.771
  • 5 - depthdevicetype 3 3.555697 6682
    7914.326 7990.326
  • 6 - devicetype 2 1.329180 6684
    7915.655 7987.655
  • 7 - depthlane 3 5.930117 6687
    7921.586 7987.586

60
  • Analysis of Deviance Table
  • Binomial model
  • Response detected
  • Terms added sequentially (first to last)
  • Df Deviance Resid. Df Resid. Dev
  • NULL 6719 8640.743
  • depth 1 483.8379 6718 8156.905
  • lane 3 2.7817 6715 8154.124
  • operator 7 69.9548 6708 8084.169
  • laneoperator 21 162.5833 6687 7921.586

61
(No Transcript)
62
9. Conclusions and recommendations
  • Conclusions and recommendations concerning the
    results to be reported
  • The detection rate depends on systematic
    influence variables (soil, type of mine, depth of
    mine, type of mine detector) and on random
    influence variables (operator, specimen of
    device).
  • Similarly, for the false alarm rate, with the
    difference that the type of mine and the depth of
    the mine are not influence variables.

63
9. Conclusions and recommendations
  • There are two possibilities to report the
    dependency of the detection rate on the
    systematic influence variables
  • report it separately for each combination of
    levels of the systematic influence variables,
    i.e. a detection rate for each type of device in
    combination with each soil, each type of mine and
    each depth of mine, or
  • average over some or all of these levels of
    influence variables. However, this average would
    be an artefact which needs an underlying
    agreement on the levels to be averaged and on the
    weight given to each of these levels in the
    average.
  • As an example one could start with an
    information on the soil of existing real mine
    fields in the form of soil 1, 2, 3, 4 with
    fraction 20, 40, 30, 10, and hence, one would
    average the detection rates for these four soils
    accordingly.

64
9. Conclusions and recommendations
65
9. Conclusions and recommendations
66
9. Conclusions and recommendations
67
9. Conclusions and recommendations
  • The experiment should be designed as an
    orthogonal design.
  • The results of the experiment should be
    interpreted by looking at the differences of
    detection or false alarm rates for different
    factor level combinations of the systematic
    factors, and not by looking onto their absolute
    values.
  • The absolute values depend on many more
    influence variables than taken into account in
    such a planned experiment.
  • However, the differences might reflect the
    situations in real demining processes.
Write a Comment
User Comments (0)
About PowerShow.com