An Answer and a Question Limits: Combining 2 results Significance: Does ??2 give ?2?

1
An Answer and a QuestionLimits Combining 2
resultsSignificance Does ??2 give ?2?

• Roger Barlow
• BIRS meeting
• July 2006

2
Revisit ??sb

• Calculator (used in BaBar) based on Cousins and
  Highland frequentist for s, Bayesian
  integration for ?? and b
• See http//www.slac.stanford.edu/barlow/java/stat
  istics2.html and C.P.C. 149 (2002) 97
• 3 different priors (uniform in ? ,1/? , ln ? )

3
Combining Limits?

• With 2 measurements
• x1.1 ? 0.1 and x1.2 ? 0.1
• the combination is obvious
• With 2 measurements
• xlt1.1 _at_ 90 CL and xlt1.2 _at_ 90 CL
• all we can say is xlt1.1 _at_ 90 CL

4
Frequentist problem

• Given N1 events with effcy ?1 , background b1
• N2 events with effcy ?2 ,
  background b2
• (Could be 2 experiments, or 2 channels in same
  experiment)
• For significance need to calculate, given source
  strength s, probability of result N1 ,N2 or
  less.

5
What does Or less mean?

• Is (3,4) larger or smaller than (2,5) ?

More
??
N2
Less
??
N1
6
Constraint

• If ?1 ?2 and b1b2 then N1N2 is sufficient.
  So cannot just take lower left quadrant as
  less.
• (And the example given yesterday is trivial)

7
Suggestion

• Could estimate s by maximising log (Poisson)
  likelihood
• ?-(?i s bi) Ni ln (?i s bi)
• Hence
• ? Ni ?i /( ?i s bi) - ?i 0
• Order results by the value of s they give from
  solving this
• Easier than it looks. For a given Ni this
  quantity is monotonic decreasing with s. Solve
  once to get sdata , explore s space generating
  many Ni sign of ? Ni ?i /( ?i sdata bi) -
  ?i tells you whether this estimated s is greater
  or less than sdata

8
Message

• This is implemented in the code Add
  experiment button (up to 10)
• Comments as to whether this is useful are welcome

9
Significance

• Analysis looking for bumps
• Pure background gives ??2old of 60 for 37 dof
  (Prob 1).
• Not good but not totally impossible
• Fit to backgroundbump (4 new parameters) gives
  better ??2new of 28
• Question Is this significant?
• Answer Yes
• Question How much?
• Answer
• Significance is?(??2 new?- ?2 old )
• ?(60-28)5.65

Schematic only!! No reference to any experimental
data, real or fictitious
Puzzle. How does a 3 sigma discrepancy become a
5 sigma discovery?
10
Justification?

• We always do it this way
• Belle does it this way
• CLEO does it this way

11
Possible Justification

• Likelihood Ratio Test
• a.k.a. Maximum Likelihood Ratio Test
• If M1 and M2 are models with max. likelihoods L1
  and L2 for the data, then 2ln(L2 / L1) is
  distributed as a ??2 with N1 - N2 degrees of
  freedom
• Provided that
• M2 contains M1 ?
• Ns are large ?
• Errors are Gaussian ?
• Models are linear ?

12
Does it matter?

• Investigate with toy MC
• Generate with Uniform distribution in 100 bins,
  ltevents/bingt100. 100 is large and Poisson is
  reasonably Gaussian
• Fit with
• Uniform distribution (99 dof)
• Linear distribution (98 dof)
• Cubic (96 dof)a0a1 x a2 x2 a3 x3
• FlatGaussian (96 dof) a0a1 exp(-0.5(x- a2)2/a3
  2)
• Cubic is linear Gaussian is not linear in a2 and
  a3

13
One experiment
Flat Gauss
Flat
linear
Cubic
14
Calculate ??2 probabilities of differences in
models
Compare linear and uniform models. 1 dof.
Probability flat Method OK
Compare flatgaussian and uniform models. 3 dof.
Probability very unflat Method invalid Peak at
low P corresponds to large ??2 i.e. false claims
of significant signal
Compare cubic and uniform models. 3 dof.
Probability flat Method OK
15
Not all parameters are equally useful
If 2 models have the same number of parameters
and both contain the true model, one can give
better results than the other.This tells us
nothing about the data
Shows ??2 for flatgauss v. cubic Same number of
parameters Flatgauss tends to be lower
Conclude ??2 does not give ?2?
16
But surely

• In the large N limit, ln L
• is parabolic in fitted
• parameters.
• Model 2 contains Model 1
• with a20 etc. So expect ln L
• to increase by equivalent of 3
• in chi squared.
• Question. What is wrong with this argument?
  Asymptopic? Different probability? Or is it
  right and the previous analysis is wrong?