Goodness of fit, confidence intervals and limits

1
Goodness of fit, confidence intervals and limits
fourth lecture

• Jorge Andre Swieca School
• Campos do Jordão, January,2003

2
References

• Statistical Data Analysis, G. Cowan, Oxford, 1998
• Statistics, A guide to the Use of Statistical
  Methods in the Physical Sciences, R. Barlow, J.
  Wiley Sons, 1989
• Particle Data Group (PDG) Review of Particle
  Physics, 2002 electronic edition.
• Data Analysis, Statistical and Computational
  Methods for Scientists and Engineers, S. Brandt,
  Third Edition, Springer, 1999

3
Limits
Tens, como Hamlet, o pavor do desconhecido? Mas
o que é conhecido? O que é que tu conheces, Para
que chames desconhecido a qualquer coisa em
especial? Álvaro de Campos (Fernando Pessoa)
Se têm a verdade, guardem-na! Lisbon Revisited,
Álvaro de Campos
4
Statistical tests
How well the data stand in agreement with given
predicted probabilities hypothesis.
null hypothesis H0
alternative
function of measured variables test statistics
error first kind significance level
error second kind
power to discriminate against H1
power
5
Neyman-Pearson lemma
H0 signal H1 background
Where to place tcut?
1-D efficiency (and purity) m-D def. of
acceptance region is not obvious
Neyman-Pearson lemma highest power (highest
signal purity) for a given significance level a
determined by the desired efficiency
region of t-space such that
6
Goodness of fit
how well a given null hypothesis H0 is compatible
with the observed data (no reference to other
alternative hypothesis)
coins N tosses, nh , nt N - nh
coin fair? H and T equal?
binomial distribution, p0.5
test statistic nh
N20, nh17
EnhNp10
7
Goodness of fit
P-value probability P, under H0, obtain a
result as compatible of less with H0 than the
one actually observed.
P0.0026
P-value is a random variable, a is a constant
specified before carrying out the test
Bayesian statistics use the Bayes theorem to
assign a probability to H0 (specify the prior
probability)
P value is often interpreted incorrectly as a
prob. to H0
P-value fraction of times on would obtain data
as compatible with H0 or less so if the
experiment (20 coin tosses) were repeated under
similar circunstances
8
Goodness of fit
optional stopping problem
Easy to identify the region of values of t with
equal or less degree of compatibility with the
hypothesis than the observed value (alternate
hypothesis p ? 0.5)
9
Significance of an observed signal
Whether a discrepancy between data and
expectation is sufficiently significant to merit
a claim for a new discovery
signal event ns, Poisson variable ?S background
event nb, Poisson variable ?b
prob. to observe n events
experiment nobs events, quantify our degree of
confidence in the discovery of a new effect
(?S?0)
How likely is to find nobs events or more from
background alone?
10
Significance of an observed signal
P(ngtnobs)1.7x10-4
Ex expect ?b0.5, nobs 5
this is not the prob. of the hypothesis ?S0 !
this is the prob., under the hypothesis ?S0, of
obtaining as many events as observed or more.
11
Significance of an observed signal
How to report the measurement?
estimate of ?
misleading

• only two std. deviations from zero
• impression that ?S is not very incompatible
• with zero

yes prob. that a Poisson variable of mean ?b
will fluctuate up to nobs or higher no prob.
that a variable with mean nobs will fluctuate
down to ?b or lower
12
Pearsons test
ni ?i
histogram of x with N bins
construct a statistic which reflects the level of
agreement between observed and expected
histograms
data
aprox. gaussian, Poisson distributed with
follow a distribution for N degrees of
freedom

• regardless of the distribution of x
• distribution free

larger larger discrepancy between data
and the hypothesis
13
Pearsons test
(rule of thumb for a good fit)
14
Pearsons test
15
Pearsons test
Poisson variable with
Before
ni dist. as multinomial with prob.
Set ntot fixed
Not testing the total number of expected and
observed Events, but only the distribution of x.
large number on entries in each bin
pi known
Follows a distribution for N-1 degrees of
freedom
In general, if m parameters estimated from data,
nd N - m
16
Standard deviation as stat. error
n observations of x, hypothesis p.d.f f(x?)
ML estimator for ?
measurement
analytic method RCF bound Monte Carlo graphical
standard deviation
repeated estimates each based on n obs.
estimator dist. centered
around true value ? and with true
estimated by and
Most practical estimators
becomes approx. Gaussian in the large sample
limit.
17
Classical confidence intervals
n obs. of x, evaluate an estimator
for a param. ?
obtained and its p.d.f. (for a
given ? unknown)
18
Classical confidence intervals
prob. for estimator to be inside the belt
regardless of ?
monotonic incresing functions of ?
19
Classical confidence intervals
Usually central confidence interval
20
Classical confidence intervals
Relationship between a conf. interval and a test
of goodness of fit
test the hypothesys using
having equal or less agreement than the
result obtained
P-value a (random variable) and ? a is
specified
Confidence interval a is specified first, a is a
random quantity depending on the data
21
Classical confidence intervals
Many experiments the interval would include the
true value in
It does not mean that the probability that the
true value of is in the fixed interval is
Frequency interpretation is not a random
variable, but the interval fluctuates since it is
constructed from data.
22
Gaussian distributed
Simple and very important application
Central limit theorem any estimator linear
function of sum of random variables becomes
Gaussian in the large sample limit.
known, experiment resulted in
23
Gaussian distributed
24
Gaussian distributed
Choose quantile
1 0.6827 1 0.8413 2 0.9544 2 0.9772 3 0.
9973 3 0.9987
Choose confidence level
0.90 1.645 0.90 1.282 0.95 1.960 0.95 1.64
5 0.99 2.576 0.99 2.326.