An Introduction to Statistical Inference

1
An Introduction to Statistical Inference

• Mike Wasikowski
• June 12, 2008

2
Statistics

• Up till now, we have looked at probability
• Analyzing data in which chance played some part
  of its development
• Two main branches
• Estimation of parameters
• Testing hypotheses about parameters
• To use statistical analysis, must ensure we have
  a random sample of the population
• Methods described are classical methods,
  "probability of data D given hypothesis H"
• Bayesian methods are also sometimes used,
  "probability of hypothesis H given data D"

3
Contents

• What is an estimator?
• Unbiased estimators
• Biased estimators
• Parametric hypothesis tests
• Nonparametric hypothesis tests
• Multiple tests/experiments

4
Classical Estimation Methods

• Probability distributions PX(x?) and density
  functions fX(x?) have parameters
• Can use the observed value x of X to estimate ?
• To estimate the parameters, must use multiple iid
  observations, x1, x2, ..., xn
• Estimator of parameters ? is a function of the
  rv's X1, X2, ..., Xn, written as either ?(X1, X2,
  ..., Xn) or ?
• Value of ? is the estimate of ?

5
Desirable Properties of Estimators

• Unbiased E(?) ?
• Small variance observed value of ? should be
  close to ?
• Normal distribution, either exactly or
  approximately allows us to use the properties of
  the normal distribution to provide properties of ?

6
Estimating µ

• Use X to estimate µ
• Know the mean value of X is µ, so X is unbiased
• Know the variance of X is s2/n, so it is small
  when n is large
• Central limit theorem tells us that the
  distribution of X will be approximately normal
  with a large number of observations
• Our estimated value of µ is x

7
Confidence Intervals

• From section 1.10.2, for large n, P(X-2s/sqrt(n)
  lt µ lt X2s/sqrt(n)) 0.95
• Probability of random interval (X-2s/sqrt(n),
  X2s/sqrt(n)) containing µ is approximately 95
• Observed value of interval given all xi is
  (x-2s/sqrt(n),x2s/sqrt(n))

8
Estimating s2

• Can develop unbiased estimator of s2 by s2
  (S(Xi-X)2)/(n-1)
• Our estimated value of s2 is s2
  (S(xi-x)2)/(n-1)
• One potential problem unless n is very large,
  this variance will also typically be large
• Variance of X s2/n (S(Xi-X)2)/(n(n-1))

9
Estimated Confidence Intervals

• We then have the 95 confidence interval for µ as
  (X-2S/sqrt(n), X2S/sqrt(n))
• Observed interval from data is (x-2s/sqrt(n),
  x2s/sqrt(n))
• Again, warning unless n is very large, this
  interval will be large and may not be useful

10
Binomial and Multinomial Probability Estimates

• Consider RV Y(p,n), where p is a parameter and n
  is the index
• Know the mean value of Y/n is p and variance of
  Y/n is p(1-p)/n
• By above, p Y/n is an unbiased estimator of p
• Typical estimate of variance of p is p(1-p)/n
  y(n-y)/n3, where y number of successes
• Above estimate is biased, unbiased estimate is
  y(n-y)/(n3-n) similarly to s2 estimate
• Estimate of pi is similarly calculated by
  converting multinomial problem into a series of
  binomial problems

11
Biased Estimators

• Not all estimators are unbiased
• Biased estimator ? is one where E(?) differs from
  ?
• Bias E(?)- ?
• Assess accuracy of ? by MSE rather than variance
• MSE(?) E((? - ?)2) Var(?)Bias(?)2
• When E(?) ?O(n-1), call the estimator
  asymptotically unbiased
• MSE and variance would differ by O(n-2)

12
Why use biased estimators?

• Some parameters cannot be estimated in an
  unbiased manner
• Biased estimators are better than unbiased
  estimators if MSE lt variance

13
Hypothesis Testing

• Test a null hypothesis (H0) versus an alternate
  hypothesis (H1 or Ha)
• Five steps
• Declare null hypothesis and alternate hypothesis
• Select significance level a
• Determine the test statistic to be used
• Determine what observed values of test statistic
  would lead to rejection of H0
• Use data to determine whether observed value of
  test statistic meets or exceeds significance
  point from step 4

14
Declaring Hypotheses

• Must declare null hypothesis and alternate
  hypothesis before seeing any data to avoid bias
• Hypotheses can be simple (specifies all values of
  unknown parameters) or complex (does not specify
  all values of unknown parameters)
• Natural alternate hypothesis is complex
• Alternate hypotheses can be either one-sided (?gt
  ?0 or ? lt ?0) or two-sided (? ! ?0)

15
Selecting Significance Level

• Two types of errors can be made from a hypothesis
  test
• Type I reject H0 when it is true
• Type II fail to reject H0 when it is false
• Unless we have limitless observations, cannot
  make the probability of making either error
  arbitrarily small
• Typical method is to focus on type I errors and
  fix a to be arbitrarily low
• Common values of a are 1 and 5

16
Choosing Test Statistic

• There is much theory available for choosing good
  test statistics
• Chapter 9 (Alex) discusses finding the optimal
  test statistic that, for a given type I error
  rate, will minimize the rate of making type II
  errors for a number of observations

17
Finding Significance Points

• Find the value of the significance points K for
  the test statistic
• General ex a 0.05, P(type I error) P(X gt K
  H0) 0.05
• If the RV is discrete, it may be impossible to
  find an exact value of K such that the rate of
  type I errors is exactly a
• In practice, we err conservatively and round up
  the value of K

18
Finding Conclusions

• Compare the result of the test statistic derived
  from observations to significance points K
• Two conclusions can be drawn from a hypothesis
  test fail to reject null, or reject null in
  favor of alternate
• A hypothesis test never tells you if a hypothesis
  is true or false

19
P-values

• An equivalent method skips calculating
  significance point K
• Instead, calculate the achieved significance
  level (p-value) of the test statistic
• Then compare p-value to a
• If p-value lt a, reject H0
• If p-value gt a, fail to reject H0

20
Power of Hypothesis Tests

• Recall step 3 involves choosing an optimal test
  statistic
• If both hypotheses are simple, choice of a
  implicitly determines ß, rate of type II error
• Power of hypothesis test 1- ß, rate of avoiding
  type II errors
• If we have a complex alternate hypothesis,
  probability of rejecting H0 depends on actual
  value of parameters in test, so there is no
  unique value of beta
• Chapter 9 discusses how to find the power of
  tests with alternate hypotheses

21
Z-test

• Classic example what is the mean of data drawn
  from a normal distribution?
• H0 µ µ0, H1 µ gt µ0
• Use X as our optimal test statistic
• RV Z (X - µ0)sqrt(n)/s has distribution N(0,1)
  when H0 is true
• For a 0.05, get Z 1.645 for significance level

22
One-sample t-test

• Must estimate the sample variance with s2
• Now use one-sample t-test, t (x-µ0)sqrt(n)/s
• If we know that X_1, X_2, ..., X_n are
  NID(mu,sigma2), H0 distribution is well known
• T (X-µ0)sqrt(n)/S
• Called the t-distribution with n-1 degrees of
  freedom
• T is asymptotically equal to Z, differs greatly
  for small n

23
Two-sample t-test

• What if we need to compare between two different
  RV's?
• Ex repeated experiment comparing two methods
• H0 µ1 µ2, H1 µ1 ! µ2
• Consider X11, X12, ..., X1m NID(µ1, s2) and
  X21, X22, ..., X2n NID(µ2,s2) to be RV's from
  which our observations are drawn
• Use two-sample t-test
• Large positive or negative values cause rejection
  of H0

24
Two-sample t-test

• T-distribution RV
• Observed value of RV

25
Paired t-test

• Suppose values of X1i and X2i are logically
  paired by some manner
• Can instead perform a paired t-test, use Di
  X1i-X2i for our test
• H0 µD 0, H1 µD ! 0
• Then use T Dsqrt(n)/SD as our test statistic
• This method can eliminate sources of variance
• Beginnings of source for ANOVA, where we break
  variation into different components
• Also foundations for F-test, test of ratio
  between two variances

26
Chi-square test

• Consider a multinomial distribution
• H0 pi specific value for each i1..k, H1 at
  least one of pi ! predefined value
• Use X2 as our test statistic, X2
  S(Yi-npi)2/(npi)
• Larger observed values of X2 will lead to
  rejection of H0
• When H0 is true and n is large, X2 chi-square
  distribution with k-1 degrees of freedom

27
Association tests

• Compare elements of a population by placing into
  one of a number of categories for two properties
• Fisher's exact test compares two different binary
  properties of a population
• H0 two properties are independent of one
  another, H1 two properties are dependent in some
  manner
• Can also use chi-square test on tables of
  arbitrary number of rows and columns

28
Hypothesis Testing with Maximum as Test Statistic

• Bioinformatics has several areas where maximum of
  many RV's is a useful test statistic
• BLAST, local alignment of sequences, only care
  about the most likely alignment
• Let X1, X2, ..., Xn N(µi,1)
• H0 µi 0 for all i, H1 one µi gt 0 with the
  rest µi 0
• Optimal test statistic Xmax
• Reject H0 if P(Xmax gt xmax H_0) lt a
• Use equation 1-F(xmax)n to find P-value
• Some options still exist if we cannot calculate
  the cdf, one possibility is total variation
  distance

29
Nonparametric Tests

• Two-sample t-test is a distribution-dependent
  test, relies on RV's having the normal
  distribution
• If we use the t-test when at least one of the
  underlying RV's is not normal, using the
  calculated p-value will result in an invalid
  testing procedure
• Nonparametric, or distribution-free, tests avoid
  problems with using tests specific to a
  distribution

30
Permutation Tests

• Avoids assumption of normal distribution
• Have RV's X11, X12, ..., X1m iid and X21, X22,
  ..., X2n iid, with possibly differing
  distributions
• Assume that X1i independent of X2j for all (i,j)
• H0 X1i's distributed identically as X2j's, H1
  distributions differ
• Q nCr(mn,m) possible placements of X11, X12,
  ..., X1m, X21, X22, ..., X2n into two groups,
  permutations
• H0 says each Q has same probability of arising

31
Permutation Tests

• Calculate test statistic for each permutation
• Reject H0 if observed value of statistic is among
  the most 100a extreme values of the test
  statistic
• Choice of test statistic depends on what we think
  may be different about the two distributions
• t-tests could be used if we feel they have
  different means, F-test if different variances
• Problems with these tests granularity with too
  few samples, computational complexity with too
  many

32
Mann-Whitney Test

• Frequently used alternative to two-sample t-test
• Observed values x11, x12,...,x1m and x21, x22,
  ..., x2n are listed in increasing order
• Associate all observations with their rank in
  this list
• Sum of all ranks is (mn)(mn1)/2
• H0 X1i's, X2j's are identically distributed, H1
  at least one parameter of the distributions
  differ
• For large sample sizes, use central limit theorem
  to test null hypothesis using z-score
• For small sample sizes, can calculate exact
  p-value as a permutation test

33
Wilcoxon Signed-rank Test

• Test for value of the median of a generic
  continuous RV if distribution is symmetric, also
  tests for mean
• H0 med M0, H1 med ! M0
• Calculate absolute differences xi- M0, order
  from smallest to largest, give ranks to each
  value
• Observed test statistic sum of ranks of
  positive differences
• Use central limit theorem to compare groups with
  large number of samples
• Can also calculate exact p-value as permutation
  for small sample sizes

34
Multiple Associated Tests

• If we test many associated hypotheses where each
  H0 is true, chance will lead to one or more being
  rejected
• Family-wide p-value can be used to avoid this
  result
• If we want a family-wide significance level of
  0.05, each test should have a 0.05/g, the
  number of different tests we are performing
• This correction applies even if the tests are not
  independent of one another, recall indicator
  variable discussion
• Obvious problem if we perform multiple different
  tests, this procedure will result in a very low
  required p-value to reject H0 for each individual
  test

35
Multiple Experiments

• In science, it is common to repeat tests to
  verify results
• What if the p-values of each test are close to a
  but not less?
• Use a combined p-value to show significance of
  each p-value in conjunction with others
• V -2log(P1P2...Pk) gives a quantity with a
  chi-square distribution of 2k degrees of freedom
• Can result in seeing significant results when no
  individual null hypothesis was rejected

36

• Questions?