Multiple comparisons

1
Multiple comparisons

• What do we do with a significant
• F ratio?

2
A priori (planned) comparisons

• A priori comparisons are planned before the data
  are collected.
• One strategy is to allow the Type I error rate
  for the study to rise, in order to maximize power
  and minimize Type II error rates.
• For that strategy to be effective, make very few
  two-group comparisons -- the ones specified in
  your research hypotheses.

3
Planned comparisons

• Another strategy for planned comparisons sets the
  a level lower than .05, in order to keep the
  experimentwise error rate at a.
• For example, with three comparisons, set a at
  .0167, so that the total Type I error rate will
  be .0167 .0167 .0167 .05
• That approach sacrifices power, and increases
  Type II error. Sound familiar?

4
Orthogonality in planned comparisons

• Some scholars insist that planned comparisons
  must be orthogonal or independent. Only k - 1
  comparisons can be orthogonal.
• Orthogonal comparisons have coefficients which
  when multiplied together sum to 0
• Contrast A 1 -1 0 S 0
• Contrast B .5 .5 -1 S 0
• Product (A x B) .5 -.5 0 S 0

5
Using t for planned comparisons

• Using either orthogonal contrast coefficients or
  a minimum number of contrasts (textbook
  approach), you could compare two groups using the
  independent samples t or ANOVA.
• t M1 - M2 M1 - M2
• SM1 -M2 SS1 SS2 1 1
• n1 n2 - 2 n1 n2
• But, since you already know MSW, use it

6
t with MSW

• t M1 - M2 M1 - M2
• MSW 1 1 sW2 1 1
• n1 n2 n1 n2
• Obviously, if n1 n2, the denominator
• simplifies to 2sW2/n

7
A posteriori tests

• Several post hoc tests have been developed to
  find the differences between means if the ANOVA F
  ratio is significant.
• Tukeys HSD keeps the total Type I error rate
  (the experimentwise error rate) at a.
• Newman-Keuls keeps the Type I error rate at a for
  each comparison.

8
Tukeys HSD

• Both Tukey and Newman-Keuls use the studentized
  range distribution, q, rather than t or F.
• qobt Mi - Mj
• MSW/n
• where i refers to the larger mean and n is the
  sample size, with all samples being the same
  size..

9
Evaluating Tukey

• Use table B-5 to find the critical value of q.
• If qobt gt qcrit, reject H0. If not, retain H0.
• Analogous to the confidence interval approach,
  you can simplify multiple Tukey HSD tests by
  determining the size of the mean difference (Mi
  Mj ) that is significant
• HSD q MSW/n
• Any mean differences that equal or exceed HSD are
  significant.

10
Newman-Keuls test

• Calculation of qobt is the same as for Tukeys
  HSD.
• The value of qcrit differs for each comparison,
  depending on r, the number of means encompassed
  by the two means being compared.
• Compare qobt to the appropriate qcrit starting
  from the largest and moving down as far as
  necessary Logic.

11
The case of unequal sample sizes

• If sample sizes are not equal, but are not
  greatly different, either, you may use Tukeys
  HSD or Newman-Keuls.
• However, what value of n should you use in the
  equation?
• Use the harmonic mean
• n k .
• (1/n1) (1/n2) (1/nk)
  

12
The Scheffe test

• This is a very conservative test, with relatively
  low power.
• F is still equal to MSB / MSW , but MSB is
  re-computed using only the two groups being
  compared.

MSB
F is compared to the same critical value as for
the ANOVA.
13
Type I Error and Power

• Which of the three approaches has the highest
  risk of Type I error?
• Which has the most power?