Title: Statistical Design of Demining Experiments and Analysis by Logistic Regression
1Statistical Design of Demining ExperimentsandAna
lysis by Logistic Regression
- Peter-Th. Wilrich
- Free University of Berlin
- Institute for Statistics and Econometrics
- ITEP Workshop
- Reliability Tests for Demining
- December, 16-17, 2003
- Federal Institute for Materials Research and
Testing (BAM)
2- Aim of the investigation
- Response variables and influence variables
- Factor settings for the experiment Oberjettenberg
(May 2003) - Experimental design of the experiment
Oberjettenberg (May 2003) - The experiments Benkovac (July 2003) and
Oberjettenberg (November 2003) - The data for analysis
- The model for analysis
- Some results of the analysis
- Conclusions and recommendations - concerning the
results to be obtained in a future
experiment- concerning the uncertainty of the
results- concerning the experimental effort
31. Aim of the investigation
- Primarily, the aim of each of the experiments was
to compare different types of devices for mine
detection with respect to their ability to detect
mines. - For this reason, two performance measures, the
detection rate and the false alarm rate, are
estimated statistically on the basis of the data
obtained in the experiments. - The detection rate DR is defined as the
probability of detecting a mine. - The false alarm rate FAR is defined as the
average number of false alarms per square meter. - Both rates do not only depend on the type of
device, but on many other factors, for instance
the soil, the operator, the type of mine, the
depth of the mine, - The dependency of these two rates, DR and FAR, on
the factors is to be estimated statistically, and
the uncertainty of the estimates has to be
reported.
42. Response variables and influence variables
52. Response variables and influence variables
63. Factor settings for the experiment
Oberjettenberg (May 2003)
- An influence variable which is included in the
experiment by setting or controlling its levels,
is called a factor. - For many factors the designer of the experiments
has to accept the levels being used in the
experiment, for others he has the freedom to
decide.
73. Factor settings for the experiment
Oberjettenberg (May 2003)
- The following factors have been included into the
experiment Oberjettenberg (May 2003) - In Oberjettenberg, the following test lanes have
been available - a lane with a slightly uncooperative soil
- three lanes with cooperative soil
- They define a qualitative factor, called lane,
i.e. it is measured on a nominal scale, with the
levels lane 1, , lane 4, which represent the
four soil types available. - It is treated as a systematic factor, i.e. we are
interested in an estimation of the effect of each
of the four soil types separately.
83. Factor settings for the experiment
Oberjettenberg (May 2003)
- 8 operators had been available, some of them
more, some less trained. - They define a qualitative factor, called
operator, with the levels operator A, ,
operator H, which represent the influence of
the operator on the response variable. - It is treated as a random factor, i.e. we are
not interested in an estimation of the effect of
these 8 operators separately. Instead we look at
these operators as being a sample out of a
population of operators, and we are interested
in the standard deviation as a measure of the
variation of the influence of the operators on
the response variable.
93. Factor settings for the experiment
Oberjettenberg (May 2003)
- Type of mine and depth of mine
- Unfortunately, the test lanes had to be accepted
as they have been, i.e. equipped with about 30
mines and other ITOPs in various depths in each
lane. -
- Hence, it was not possible to set the types of
mines and the depths as levels of factors.
However, since the information was available,
these influence variables have been included in
the analysis (with difficulties in the
interpretation of the results).
10- depth cm 1 2 3 4 5 6 7 8 10 12 15 16 17 20
22 - 100Cr6 - 16mm 0 0 0 0 0 0 0 0 0 0 0 0 0 2
1 - Maus 4 6 2 1 3 0 0 0 0 0 0 0 0 0
0 - O0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
0 - PMN 0 3 3 0 2 0 0 0 0 0 3 0 1 0
0 - PPM-2 0 5 4 2 1 0 0 0 0 0 0 0 0 0
0 - PT-Mi-Ba-III 0 0 0 0 2 1 1 2 0 0 0 0 0 0
0 - SchAMi DM 31 0 0 0 0 2 1 1 0 0 0 0 0 0 0
0 - TM-46 0 0 0 0 2 1 0 1 0 0 0 0 0 0
0 - TM-62 M 0 0 0 0 1 2 1 0 0 0 2 1 1 0
0 - TM-62 P2 0 0 0 0 2 1 0 0 0 0 0 0 0 0
0 - TM-62 P3 0 0 0 0 3 2 2 0 0 0 0 0 0 0
0 - large G0 0 0 0 0 0 0 0 0 3 1 0 0 0 0
0 - large I0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
0 - large K0 0 0 0 0 0 0 0 0 2 1 0 0 0 0
0 - small C0 0 0 0 0 3 0 1 0 0 0 0 0 0 0
0 - small E0 0 0 0 0 0 0 0 0 2 0 0 0 0 0
0 - small G0 0 0 0 0 2 0 1 0 0 0 0 0 0 0
0
113. Factor settings for the experiment
Oberjettenberg (May 2003)
- Type of the mine detection device
- It was decided to include 4 mine detection
devices into the experiment. - They define a qualitative factor, called
devicetype, with levels devicetype 1, ,
devicetype 4. - It is treated as a systematic factor since we
are interested to estimate the effect of each of
the device types on the response variable is our
primary interest.
123. Factor settings for the experiment
Oberjettenberg (May 2003)
- Specimen of the mine detection device
- For each of the 4 mine detection devices two
specimens have been asked for, in order to get to
know the variation between specimens of one and
the same device type, expressed as its component
of standard deviation. - This is a qualitative factor, called specimen,
with two levels specimen 1 and specimen 2. - It is treated as a random factor, nested in the
factor devicetype, because the level specimen
1 or specimen 2 is meaningful only for a
particular type of device but not across the
factor devicetype.
133. Factor settings for the experiment
Oberjettenberg (May 2003)
- Sensibility of the device
- It was decided to use each specimen of each type
of device on a high level and on a low level of
sensibility. - This defines a qualitative factor, called
sensibility, with the two levels low and
high. - It is treated as a systematic factor.
-
- This concludes the list of factors having been
investigated in the Oberjettenberg (May 2003)
experiment.
144. Experimental design for the experiment
Oberjettenberg (May 2003)
- A full factorial design consists of obtaining at
least one value of the response variable for each
factor level combination. - There are 4 x 8 x 4 x 2 x 2 512 factor level
combinations. - Without any repetitions, this would mean 512
runs of operators at these lanes, or 1024 with
one repetition. - Such a large experiment did not fit into the
budget and time frame and it is not necessary
to estimate the two rates sufficiently accurate. - Hence, a suitable number of runs has to be
selected out of these 512 factor level
combinations.
154. Experimental design for the experiment
Oberjettenberg (May 2003)
- A basic requirement is that the main effects of
all factors shall be estimated uncorrelated with
the estimates of other main effects. - Remembering the table of types of mines and depth
of mines of the test fields in Oberjettenberg, it
is immediately clear that this is not the case
for these two influence variables. Some mine
types are in these test fields only at one (or
two different) depth, and on the other hand
some depths are only connected with one (or two)
mine types. If, with this experimental design as
the basis, the estimated detection rate for a
particular depth is reported, one could it as
well interpret as the detection rate for a
particular type of mine. - An uncorrelated estimation of all main effects
(but not of all interaction effects) can be
achieved if an orthogonal design is used. Here, a
design based on a Graeco Latin square design, was
used.
164. Experimental design for the experiment
Oberjettenberg (May 2003)
- Deminers A, B, C, D, E, F, G, H
- Devices a, a, b, b, g, g, d, d
175. The experiments Benkovac and Oberjettenberg
(November 2003)
- The experimental designs of the experiments
Benkovac and Oberjettenberg (November 2003) are
similar to the experimental design of
Oberjettenberg (May 2003). - In the Benkovac experiment types of mines (and
other targets) and depths of mines were included
as factors with levels set so that the effects of
these two factors are not (or almost not)
confounded so that they can be estimated
uncorrelated with each other.
18- Benkovac (July 2003)
- depthcm 0 5 10 13 20
- E0 0 4 0 0 0
- G0 0 4 0 0 0
- K0 0 4 0 0 0
- PMA-1A 4 12 4 4 4
- PMA-2 4 12 4 4 4
- PMA-3 4 12 4 4 4
- PROM-1 8 12 0 0 0
- TMA-3 0 0 1 0 0
- TMA-4 0 0 3 0 0
- TMM-1 0 0 1 0 0
- TMRP-6 0 0 3 0 0
- ball 0 0 4 0 0
195. The experiments Benkovac and Oberjettenberg
(November 2003)
- The Oberjettenberg (November 2003) actually
consists of two experiments - one is a replication of the Oberjettenberg (May
2003) experiment with some slight changes
concerning the operators and the sensibility
level. - For the other one the test lanes have been newly
equipped with mines (and other targets) as
follows
20- Oberjettenberg (November 2003)
- depthcm 0 5 10 13 20
- ball 0 0 3 0 0
- C0 0 3 0 0 0
- E0 0 3 0 0 0
- G0 0 3 0 0 0
- I0 0 3 0 0 0
- K0 0 3 0 0 0
- PMA-S 3 9 3 3 3
- Maus 3 9 3 3 3
- PMNMS3 3 9 3 3 3
- TM-62 M 0 0 3 0 0
216. The data for analysis
- The first data frame, to be used for the
estimation of the detection rate, OJT1, has 6720
rows (one for each situation in which a mine can
be detected or not) and 11 columns (one for each
variable). The following table shows the first 15
and the last 15 rows of this data frame.
22(No Transcript)
236. The Data for Analysis
- The second data frame, to be used for the
estimation of the false alarm rate, OJF1, has 256
rows (one for each start of an operator) and 8
columns (one for each variable). The following
table shows the first 15 and the last 15 rows of
this data frame.
24(No Transcript)
257. The model for analysis
267. The model for analysis
277. The model for analysis
287. The model for analysis
297. The model for analysis
307. The model for analysis
317. The model for analysis
- The new model is the special case of logistic
regression in the class of generalized linear
models introduced by Nelder and Wedderburn in
1972 and treated in a monograph by McCullagh and
Nelder in 1983. - We overcome all difficulties with the model
assumptions and we solve problem 2., i.e. we can
include mine type and mine depth into our
determination of the detection rate, even in our
case where these two influence variables have not
been taken into account as factors. - However, we cannot any more estimate the
parameters by the method of least squares.
Instead, we have to use the method of Maximum
Likelihood and it would not be possible to find
a solution without a computer.
32(No Transcript)
33(No Transcript)
34(No Transcript)
35(No Transcript)
367. The model for analysis
378. Some results of the analysis
- The analysis with the generalized linear models
was carried out for the four experiments - Oberjettenberg (May 2003)
- Benkovac (July 2003)
- Oberjettenberg (November 2003), part 1
- Oberjettenberg (November 2003), part2
38(No Transcript)
39(No Transcript)
40(No Transcript)
41(No Transcript)
42(No Transcript)
43(No Transcript)
44(No Transcript)
45(No Transcript)
46(No Transcript)
47(No Transcript)
48(No Transcript)
49(No Transcript)
50(No Transcript)
51(No Transcript)
52(No Transcript)
53(No Transcript)
54(No Transcript)
55(No Transcript)
56(No Transcript)
57(No Transcript)
58- Analysis of Deviance Table
- Binomial model
- Response detected
- Terms added sequentially (first to last)
- Df Deviance Resid. Df Resid.
Dev - NULL 6719
8640.743 - depth 1 483.8379 6718
8156.905 - devicetype 3 32.7918 6715
8124.114 - lane 3 2.7913 6712
8121.322 - operator 7 70.8227 6705
8050.500 - depthdevicetype 3 2.5687 6702
8047.931 - depthlane 3 6.3224 6699
8041.609 - depthoperator 7 7.7671 6692
8033.841 - devicetypelane 8 2.6677 6684
8031.174 - devicetypeoperator 18 11.4115 6666
8019.762 - laneoperator 12 124.7781 6654
7894.984
59- Analysis of Deviance Table
- Initial Model
- detected (depth devicetype lane
operator)2 - Final Model
- detected depth lane operator
laneoperator - Step Df Deviance Resid. Df
Resid. Dev AIC - 1 6654
7894.984 8026.984 - 2 - devicetypeoperator 12 4.731628 6666
7899.716 8007.716 - 3 - devicetypelane 6 1.399623 6672
7901.115 7997.115 - 4 - depthoperator 7 9.655222 6679
7910.771 7992.771 - 5 - depthdevicetype 3 3.555697 6682
7914.326 7990.326 - 6 - devicetype 2 1.329180 6684
7915.655 7987.655 - 7 - depthlane 3 5.930117 6687
7921.586 7987.586
60- Analysis of Deviance Table
- Binomial model
- Response detected
- Terms added sequentially (first to last)
- Df Deviance Resid. Df Resid. Dev
- NULL 6719 8640.743
- depth 1 483.8379 6718 8156.905
- lane 3 2.7817 6715 8154.124
- operator 7 69.9548 6708 8084.169
- laneoperator 21 162.5833 6687 7921.586
61(No Transcript)
629. Conclusions and recommendations
- Conclusions and recommendations concerning the
results to be reported - The detection rate depends on systematic
influence variables (soil, type of mine, depth of
mine, type of mine detector) and on random
influence variables (operator, specimen of
device). - Similarly, for the false alarm rate, with the
difference that the type of mine and the depth of
the mine are not influence variables.
639. Conclusions and recommendations
- There are two possibilities to report the
dependency of the detection rate on the
systematic influence variables - report it separately for each combination of
levels of the systematic influence variables,
i.e. a detection rate for each type of device in
combination with each soil, each type of mine and
each depth of mine, or - average over some or all of these levels of
influence variables. However, this average would
be an artefact which needs an underlying
agreement on the levels to be averaged and on the
weight given to each of these levels in the
average. - As an example one could start with an
information on the soil of existing real mine
fields in the form of soil 1, 2, 3, 4 with
fraction 20, 40, 30, 10, and hence, one would
average the detection rates for these four soils
accordingly.
649. Conclusions and recommendations
659. Conclusions and recommendations
669. Conclusions and recommendations
679. Conclusions and recommendations
- The experiment should be designed as an
orthogonal design. - The results of the experiment should be
interpreted by looking at the differences of
detection or false alarm rates for different
factor level combinations of the systematic
factors, and not by looking onto their absolute
values. - The absolute values depend on many more
influence variables than taken into account in
such a planned experiment. - However, the differences might reflect the
situations in real demining processes.