Group comparisons: Intro to factorial design

1
Group comparisonsIntro to factorial design

• 3143/Josh Rodefer, Ph.D.

2
Previously (?)

• Experimental design
• Design Between vs Within
• Compare 2 groups (t test different types)
• Paired (within repeated measures)
• Unpaired (between)
• Control confounds
• Extraneous variables (random vs systematic)
• How to control or minimize?

3
Basic issues in experimental

• What do you need for an experiment?
• An independent variable (what the researcher
  varies minimally 2 groups)
• Equal assignment to groups (lots of ways to do
  this)
• Controlling extraneous variables (alternative
  explanations confounds)

4
Hypothesis testing 2 samples

• Want to test relationship between the means
• Use a t-test
• Used when you have 2 levels of 1 IV
• Dependent samples (1 sample, tested twice)
• Independent samples (2 different samples)
• aka Within- (dependent) and Between (independent)
  designs

5
Hypothesis testing 2 samples

• Dependent samples (Within)
• Eg, before and after treatment (or pre/post)
• 20 people tested on attitude towards recycling
  after watching nature video
• IVviewing nature video
• Level 1 test score pre-video
• Level 2 test score post-video
• DV score on test
• 2 samples, same people in each sample

6
Hypothesis testing 2 samples

• Independent samples (between)
• 2 scores are taken, but from different groups 2
  independent samples (random)
• Eg, study effects of age on swearing
• 20 ten yr olds and 20 fifteen yr olds, record
  swearing in 2 hr period
• IVage
• Level 1 10 yrs old
• Level 2 15 yrs old

7
More than 2 groups ANOVA

• Analysis of Variance
• Tests the differences between treatment groups
  (conditions) to see if they are significant
• Look at variance in DV, partition it into 2
  components
• variance due to IV (good)
• variance due to error (bad extraneous variables)
• Asks if the ratio (IV variance/error variance)
  between the two types of variance is greater than
  would be expected due to chance (or equal or
  about 1.00)
• F test examines this ratio

8
ANOVA when to use?

• One IV at least 3 conditions
• Between subjects One-way ANOVA
• Eg, effect of psychiatric diagnosis (depression,
  panic, no disorder) on memory
• Within subjects Repeated measures ANOVA
• Eg, experiment where subjects experience all
  conditions example Stroop effect, colors/words
• Two (or more) IVs
• Factorial ANOVA
• Effect of psychiatric diagnosis on Stroop effect
  multiple IVs

9
First One way ANOVA

• One IV 3 (or more conditions)
• Single test
• Tests Null that all group means are equal
• Alternative all group means are not equal
• Null must be non-directional (vs t test
  directional)
• Between subjects ANOVA
• Analogous to independent samples t-test
• (stats trivia In fact.if you have 2 independent
  groups, then Ft2)

10
One way ANOVA

• Why not just perform 3 t-tests?
• Psychiatric diagnosis memory example
• Depression vs panic disorder
• Depression vs no disorder
• Panic disorder vs no disorder
• Same thing, right?

11
One way ANOVA

• Multiple tests result in an inflated alpha
• Alpha for each test 0.05 (risk for Type I)
• If you do 3 tests, then each is at 0.05 and they
  are additive
• Thus, doing 3 t-test increases the
  experiment-wise Type I error rate from 0.05 to
  0.15 (not acceptable)
• Must use ANOVA to test all combinations
  simultaneously, keeping alpha at 0.05

12
One way ANOVA

• So, we can calculate the total variance and
  determine how much is due to IV and how much is
  error
• F ratio variance due to IV /error variance
• Variance due to IV is called MSB (Mean square
  between groups)
• Variance due to extraneous variables or error is
  called MSW (Mean square within groups)
• F test formula F MSB/MSW

13
ANOVA

• Most data sets have both error variance and
  variance due to the IV (or, both between and
  within group variability)
• We want to know if the between group variance is
  due to a true effect of the IV (is this effect
  real?)
• FMSb/MSw
• If F about 1, then no effect of IV
• If F 1 then may have an effect of IV (need to
  look up value on F table, depends on critical F
  and df)

14
Factorial ANOVA 2 way designs

• Have 2 (or more) IVs in the same experiment eg,
  test effects of gender and age on humor
• IV gender (M/F)
• IV age (10, 20, or 30 yrs old)
• DV freq of laughing during comedy show
• Why not conduct 2 separate experiments?
• Gender on humor age on humor

15
Factorial ANOVA 2 way designs

• Why do you want to look at 2 IVs?
• More efficient study factor A then study factor
  Boryou could study them together
• More interesting can see how things relate
  together
• More representative of the real world

16
Factorial designs

• Have 2 (or more) IVs
• Each IV is called a factor
• Each factor has at least 2 levels/conditions
• The design must be complete each level of one IV
  must occur with each level of the other IV(s)
• Described as 2x2 or 3x5 or2x3x4
• Terminology Factors/Main effects levels cells
• Can be experiment or quasi-experiment (whats
  quasi?)

17
Factorial designs

• Example examine effect of sleep and caffeine on
  reaction time study
• IV-1 Sleep 2 hrs vs 8 hrs
• IV-2 Caffeine zero (water) vs 3 Cokes
• DV computer game reaction time
• Both IVs are between subjects variables

18
Factorial design

• Randomly select subject for each condition equal
  cell sizes
• 2 IVs test for 3 effects
• 2 Main effects (effects of sleep and caffeine)
• 1 interaction (interaction of sleep caffeine)
• Thus, 3 null hypotheses, 3 alternative
  hypotheses, and 3 F tests

19
Factorial designs

• Main effect
• The effect of one factor (IV) collapsed across
  levels of the other factor (IV) (ie, you ignore
  the other IV)
• Test one main effect for each factor (IV)
• Need to analyze statistics to check for
  significance

20
Factorial design Main effect graphic
8 hrs
2 hrs
0 Cokes
3 Cokes
Did different amounts of sleep have an effect?
Mean of 2 hrs
Mean of 8 hrs
21
Factorial design Main effect graphic
8 hrs
2 hrs
0 Cokes
Mean of 0 Cokes
3 Cokes
Mean of 3 Cokes
Did different quantities of Coca-cola have an
effect?
22
Post hoc tests

• Post hoc means after the fact
• Tests that are performed after finding a
  significant effect (sig F value)
• ANOVAs tell you if there is a difference, but
  they dont tell you where it is
• Post hoc tests answer that question

23
Post hoc tests

• Would you need a follow up post hoc test after a
  significant t-test?
• Would you need to follow up a significant F-test
  where there are 3 levels of the IV with a post
  hoc test (say, for the sleep experiment you used
  2, 4 or 8 hrs of sleep as three different levels
  of the IV).

24
Post hoc tests

• So, need to use post hoc tests when
• You have 3 or more means
• You have a significant difference with a
  general/omnibus test (F test)
• Post hoc tests (pairwise comparisons) examine all
  possible combinations of means

25
Post hoc tests

• Many types Tukeys HSD (honestly significant
  difference), Bonferroni, Scheffes,
  Kruskal-Wallis, Dunnetts
• Each tests pairwise hypotheses
• Each calculates a particular statistic that
  resembles a t-test (but it isnt that simple)

26
Post hoc tests

• Consider the 3 level IV, where you find a
  significant F value
• Next need to assess all possible pairwise
  comparison
• Mean 1 /diff mean 2
• Mean 1 /diff mean 3
• Mean 2 /diff mean 3
• Compute pairwise comparison for each
  testascertain where the difference(s) are
  statistically significant

27
Next time

• More factorials interactions

28
Previously -- 2nd Lecture

• Experimental design (Between vs Within)
• Compare 2 groups (paired vs unpaired)
• Control confounds (random vs systematic)
• ANOVAs (3 or more groups)
• Do overall F tests
• Pairwise compartions (post hoc tests)
• (like t tests done after significant F tells
  you where the difference is that is, which two
  means are significantly different from each other)

29
ANOVA when to use?

• One IV at least 3 conditions
• Between subjects One-way ANOVA
• Eg, effect of psychiatric diagnosis (depression,
  panic, no disorder) on memory
• Within subjects Repeated measures ANOVA
• Eg, experiment where subjects experience all
  conditions example Stroop effect, colors/words
• Two (or more) IVs
• Factorial ANOVA
• Effect of psychiatric diagnosis on Stroop effect
  multiple IVs

30
Factorial designs

• Have 2 (or more) IVs
• Each IV is called a factor
• Each factor has at least 2 levels/conditions
• The design must be complete (sometimes called
  completely crossed) each level of one IV must
  occur with each level of the other IV(s)
• Described as 2x2 or 3x5 or2x3x4
• Can be experiment or quasi-experiment

31
Factorial designs

• Example examine effect of sleep and caffeine on
  reaction time study
• IV-1 Sleep 2 hrs vs 8 hrs
• IV-2 Caffeine zero (water) vs 3 Cokes
• DV computer game number correct
• Both IVs are between subjects variables

32
Factorial design

• Randomly select subject for each condition equal
  cell sizes
• 2 IVs test for 3 effects
• 2 Main effects (effects of sleep and caffeine)
• 1 interaction (interaction of sleep caffeine)
• Thus, 3 null hypotheses, 3 alternative
  hypotheses, and 3 F tests

33
Factorial designs

• Main effect
• The effect of one factor (IV) collapsed across
  levels of the other factor (IV) (ie, you ignore
  the other IV)
• Test one main effect for each factor (IV)

34
Factorial designs

• Interactions are a way to qualify findings
• Revisit coke and sleep study

8 hrs
2 hrs
0 Cokes
3 Cokes
35
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
36
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
What does the parallel shift suggest to you?
37
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
38
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
Spreading interaction (aka ordinal) usually at
least one main effect
39
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
But what about if we plotted this differently?
40
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
41
Factorial Design

• Calculate cell means and plot

2 hrs
8 hrs
0 Cokes
3 Cokes
Cross-over interaction (aka disordinal) opposite
effects of IV1 at different levels of IV2
usually no main effects, but still can have
interaction (!)
42
Interactions

• So look at main effects
• Look for interactions
• Graphing things helps
• parallel lines no interactions
• crossing or close to crossing lines interaction
• But ultimately you need stats and p values to
  make judgement about significance

43
Factorials

• So with 2 Independent variables you can have
  multiple types of designs
• Both IV are within
• Both IV are between
• One IV is within and one IV is between
• Also can have 1 true IV, and 1 quasi-IV
  (sometimes called mixed or experi-corr)
• Lets you consider unmanipulated factors like
  subject variables (sex, age, diagnoses, etc.)
• The more correlational something is, the more
  external validity (and less internal validity)
  you have

44
Within designs

• Simple 2 group comparison or ANOVA, doesnt
  matter.
• Limitations?
• How to fix?

45
Counter balancing schemes

• Reverse order of group presentation (AB vs BA)
  but remember ANOVA usually have more conditions
• Gets complicated quickly.
• Consider a within subjects design IV that has 4
  levels, how many different orders of conditions?

46
Counter balancing schemes

• Reverse order of group presentation (AB vs BA)
  but remember ANOVA usually have more conditions
• Gets complicated quickly.
• Consider a within subjects design IV that has 4
  levels, how many different orders of conditions?
• ABCD

47
Counter balancing schemes

• Reverse order of group presentation (AB vs BA)
  but remember ANOVA usually have more conditions
• Gets complicated quickly.
• Consider a within subjects design IV that has 4
  levels, how many different orders of conditions?
• ABCD, ABDC,

48
Counter balancing schemes

• Reverse order of group presentation (AB vs BA)
  but remember ANOVA usually have more conditions
• Gets complicated quickly.
• Consider a within subjects design IV that has 4
  levels, how many different orders of conditions?
• ABCD, ABDC, ACBD,

49
Counter balancing schemes

• Reverse order of group presentation (AB vs BA)
  but remember ANOVA usually have more conditions
• Gets complicated quickly.
• Consider a within subjects design IV that has 4
  levels, how many different orders of conditions?
• ABCD, ABDC, ACBD, ACDB, ADBC, ADCB
• BCDA, BCAD, BDAC, BDCA, BACD, BADC
• CDAB, CDBA, CABD, CADB, CBAD, CBDA
• DABC, DACB, DBAC, DBCA, DCAB, DCBA

50
Counterbalancing

• Too confusingmore simple way is a Latin Square
  design
• Number conditions number of orders
• Each condition appears at each position only once

C
D
A
B
1st condition
2nd condition
3rd condition
4th condition
51
Fancy control groups

• Might be possible that the IV isnt all that is
  involved in a particular effect
• Other factors (timing, duration, pairing with
  other events) may be critical
• How to control? A Yoked control group
• The yoked control subject is paired directly with
  an experimental subject (matching of experiences
  in a way)

52
Summary

• How many IVs?
• 1 IV, 2 group/levels t-test
• Paired (subjects serve in both conditions) vs
  Unpaired (different subjects in both conditions)
• 1 IV, 2 groups/levels one-way ANOVA
• 2 IV two-way ANOVA (factorial design)
• Do subjects serve in only one condition?
• Yes between-subjects design (aka independent)
• No within-subjects design (aka repeated
  measures, aor dependent)
• Yes and No mixed between/within combo design
• Main effects (did the IV have an effect?)
  interactions (was the effect dependent upon
  different levels of the other IV?)