Statistical Inference June 30-July 1, 2004

1
Statistical InferenceJune 30-July 1, 2004
2
Statistical Inference The process of making
guesses about the truth from a sample.
3

• FOR EXAMPLE Whats the average weight of all
  medical students in the US?
• We could go out and measure all US medical
  students (gt65,000)
• Or, we could take a sample and make inferences
  about the truth from our sample.

Using what we observe, 1. We can test an a priori
guess (hypothesis testing). 2. We can estimate
the true value (confidence intervals).
4
Statistical Inference is based on Sampling
Variability

• Sample Statistic we summarize a sample into one
  number e.g., could be a mean, a difference in
  means or proportions, or an odds ratio  
• E.g. average blood pressure of a sample of 50
  American men
• E.g. the difference in average blood pressure
  between a sample of 50 men and a sample of 50
  women
• Sampling Variability If we could repeat an
  experiment many, many times on different samples
  with the same number of subjects, the resultant
  sample statistic would not always be the same
  (because of chance!).
• Standard Error a measure of the sampling
  variability (a function of sample size).

5
Sampling Variability
Random students
The Truth (not knowable)
The average of all 65,000 US medical students
at this moment is exactly 150 lbs
6
Sampling Variability
Random samples of 5 students
The Truth (not knowable)
The average of all 65,000 US medical students
at this moment is exactly 150 lbs
7
Sampling Variability
Samples of 50 students
The Truth (not knowable)
146.9 lbs
The average of all 65,000 US medical students
at this moment is exactly 150 lbs
148.9 lbs
150.0 lbs
152.3 lbs
147.2 lbs
155.3 lbs
8
Sampling Variability
Samples of 150 students
The Truth (not knowable)
150.31 lbs
The average of all 65,000 US medical students
at this moment is exactly 150 lbs
150.02 lbs
149.8 lbs
149.95 lbs
150.3 lbs
150.9 lbs
9
The Central Limit Theorem how sample statistics
vary

•  Many sample statistics (e.g., the sample
  average) follow a normal distribution
• centers around the true population value (e.g.
  the true mean weight)
• Becomes less variable (by a predictable amount)
  as sample size increases
• Standard error of a sample statistic standard
  deviation / square root (sample size)
• Remember standard deviation reflects the average
  variability of the characteristic in the
  population

10
The Central Limit TheoremIllustration

• I had SAS generate 1000 random observations from
  the following probability distributions
• N(10,5)
• Exp(1)
• Uniform on 0,1
• Bin(40, .05)

11
N(10,5)
12
Uniform on 0,1
13
Exp(1)
14
Bin(40, .05)
15
The Central Limit TheoremIllustration

• I then had SAS generate averages of 2, averages
  of 5, and averages of 100 random observations
  from each probability distributions
• (Refer to end of SAS LAB ONE, which we will
  implement next Wednesday, July 7)

16
N(10,25) average of 1(original distribution)
17
N(10,25) 1000 averages of 2
18
N(10,25) 1000 averages of 5
19
N(10,25) 1000 averages of 100
20
Uniform on 0,1 average of 1(original
distribution)
21
Uniform 1000 averages of 2
22
Uniform 1000 averages of 5
23
Uniform 1000 averages of 100
24
Exp(1) average of 1(original distribution)
25
Exp(1) 1000 averages of 2
26
Exp(1) 1000 averages of 5
27
Exp(1) 1000 averages of 100
28
Bin(40, .05) average of 1(original
distribution)
29
Bin(40, .05) 1000 averages of 2
30
Bin(40, .05) 1000 averages of 5
31
Bin(40, .05) 1000 averages of 100
32
The Central Limit Theorem formally

• If all possible random samples, each of size n,
  are taken from any population with a mean ? and a
  standard deviation ?, the sampling distribution
  of the sample means (averages) will

3. be approximately normally distributed
regardless of the shape of the parent population
(normality improves with larger n)
33
Example

• Pretend that the mean weight of medical students
  was 128 lbs with a standard deviation of 15 lbs

34
Hypothetical histogram of weights of US medical
students (computer-generated)
mean 128 lbs standard deviation 15 lbs
35
Average weights from 1000 samples of 2
36
Average weights from 1000 samples of 10
37
Average weights from 1000 samples of 120
38
Using Sampling Variability

• In reality, we only get to take one sample!!
• But, since we have an idea about how sampling
  variability works, we can make inferences about
  the truth based on one sample.

39
Hypothesis Testing
40
Hypothesis Testing

• The null hypothesis is the straw man that we
  are trying to shoot down.
• Example 1 Possible null hypothesis mean weight
  of medical students 128 lbs
• Lets say we take one sample of 120 medical
  students and calculate their average weight.

41
Expected Sampling Variability for n120 if the
true weight is 128 (and SD15)
42
P-value associated with this experiment
P-value (the probability of our sample average
being 143 lbs or more IF the true average weight
is 128) lt .0001 Gives us evidence that 128 isnt
a good guess
43
Estimation (a preview)
Wed estimate based on these data that the
average weight is somewhere closer to 143 lbs.
And we could state the precision of this estimate
(a confidence intervalto come later)
44
Expected Sampling Variability for n2
45

Expected Sampling Variability for n2
P-value 11 i.e. about 11 out of 100 average
of 2 experiments will yield values 143 or higher
even if the true mean weight is only 128
46
The P-value

• P-value is the probability that we would have
  seen our data (or something more unexpected) just
  by chance if the null hypothesis (null value) is
  true.
• Small p-values mean the null value is unlikely
  given our data.

47
The P-value

• By convention, p-values of lt.05 are often
  accepted as statistically significant in the
  medical literature but this is an arbitrary
  cut-off.
• A cut-off of plt.05 means that in about 5 of 100
  experiments, a result would appear significant
  just by chance (Type I error).

48
What factors affect the p-value?

• The effect size
• Variability of the sample data
• Sample size

49
Statistical Power

• Note that, though we found the same sample value
  (143 lbs) in our 120-student sample and our
  2-student sample, we only rejected the null (and
  concluded that med students weigh more on average
  than 128 lbs) based on the 120-student sample.
• Larger samples give us more statistical power

50
Hypothesis Testing example 2

• Hypothesis more babies born in November (9
  months after Valentines Day)
• Empirical evidence Our researcher observed that
  6/19 kids in one classroom had November birthdays.

51
Hypothesis Testing

• Is a contest between
• The Null Hypothesis and the Alternative
  Hypothesis
• The null hypothesis (abbreviated H0) is usually
  the hypothesis of no difference
• Example There are no more babies born in
  November (9 months after Valentines Day) than
  any other month
• The alternative hypothesis (abbreviated Ha)
• Example There are more babies born in November
  (9 months after Valentines Day) than in other
  months

52
The Steps

• 1. Define your null and alternative hypotheses
• H0 P(being born in November)1/12
• Ha P(being born in November)gt1/12

53
The Steps

• 2. Figure out the null distribution
• If I observe a class of 19 students and each
  student has a probability of 1/12th of being born
  in November
• Sounds BINOMIAL!
• In MATH-SPEAK Class binomial (19, 1/12th)
• If the null is true, how many births should I
  expect to see?
• Expected November births 19(1/12) 1.5 why?
• Reasonable Variability 19(1/12)(11/12)1/2
  1.2 why?
• If I see 0-3 November births, it seems reasonable
  that the null is trueanything else is suspicious

54
The Steps

• 3. Observe (experimental data)
• We see 6/19 babies were born in November in this
  case.

55
The Steps

• 4. Calculate a p-value and compare to a preset
  significance level

56
The Almighty P-Value

• The P-value roughly translated is the
  probability of seeing something as extreme as you
  did due to chance alone
• Example The probability that we would have seen
  6 or more November births out of 19 if the
  probability of a random child being born in
  November was only 1/12.

Easy to Calculate in SAS data _null_ pval 1-
CDF('BINOMIAL',5, (1/12), 19) put
pval run 0.003502582
57
The Steps

• 4a. Calculate a p-value
• data _null_
• pval 1- CDF('BINOMIAL',5, (1/12), 19)
• put pval
• run
• 0.003502582
• b. and compare to a preset significance level.
• .0035lt.05

58
The Steps

• 5. Reject or fail to reject (accept) Ho.
• In this case, reject Ho.

59
Summary The Underlying Logic
Follows this logic Assume A. If A, then
B. Not B. Therefore, Not A. But throw in a bit
of uncertaintyIf A, then probably B
60
Summary It goes something like this

• The assumption The probability of being born in
  November is 1/12th.
• If the assumption is true, then it is highly
  likely that we will see fewer than 6
  November-births (since the probability of seeing
  6 or more is .0035, or 3-4 times out of 1000).
• We saw 6 November-births.
• Therefore, the assumption is likely to be wrong.

61
Example 3 the odds ratio

• Null hypothesis There is no association between
  an exposure and a disease (odds ratio1.0).

62
Example 3 Sampling Variability of the null Odds
Ratio (OR) (100 cases/100 controls/10 exposed)
63
The Sampling Variability of the natural log of
the OR (lnOR) is more Gaussian
64
Statistical Power

• Statistical power here is the probability of
  concluding that there is an association between
  exposure and disease if an association truly
  exists.
• The stronger the association, the more likely we
  are to pick it up in our study.
• The more people we sample, the more likely we are
  to conclude that there is an association if one
  exists (because the sampling variability is
  reduced).

65
Error and Power

• Type-I Error (false positive)
• Concluding that the observed effect is real when
  its just due to chance.
• Type-II Error (false negative)
• Missing a real effect.
• POWER (the flip side of type-II error)
• The probability of seeing a real effect.

66
Think ofPascals Wager
67
Type I and Type II Error in a box
68
Statistical vs. Clinical Significance
Consider a hypothetical trial comparing death
rates in 12,000 patients with multi-organ failure
receiving a new inotrope, with 12,000 patients
receiving usual care. If there was a 1
reduction in mortality in the treatment group
(49 deaths versus 50 in the usual care group)
this would be statistically significant (plt.05),
because of the large sample size. However, such
a small difference in death rates may not be
clinically important.
69
Confidence Intervals (Estimation)
70
Confidence Intervals (Estimation)

• Confidence intervals dont presuppose a null
  value.
• Shows our best guess at the plausible range of
  values for the population characteristic based on
  our data.
• The 95 confidence interval contains the true
  population value approximately 95 of the time.

71
95 CI should contain true value 19/20 times
X TRUE VALUE (--------------------X---
--------------) (--------
X-------------------------) (--------------------
-X----------------)
X (-----------------------------------)
(-----------------X----------------) (----------
------------X----------------) (----X-----------
----------------------)
72
Confidence Intervals

• (Sample statistic) ? (measure of how confident
  we want to be) ? (standard error)

73
95 CI from a sample of 120143 /- 2 x (1.37)
140.26 --145.74
74
95 CI from a sample of 10143 /- 2 x (4.74)
133.52 152.48
75
99.7 CI from a sample of 10143 /- 3 x (4.74)
128.78 157.22
76
What Confidence Intervals do

• They indicate the un/certainty about the size
  of a population characteristic or effect. Wider
  CIs indicate less certainty.
•   Confidence intervals can also answer the
  question of whether or not an association exists
  or a treatment is beneficial or harmful.
  (analogous to p-values)
• e.g., if the CI of an odds ratio includes the
  value 1.0 we cannot be confident that exposure is
  associated with disease.