Experimental and Quasi-experimental Designs

1
Chapter 11

• Experimental and Quasi-experimental Designs
• Dr. Bill Bauer

2
Key Ideas

• History of experimental resign
• Key characteristics of experimental design
• Types of experimental design
• Steps in conducting experimental research
• Criteria for evaluating experimental research

3
A Brief History of Experimental Designs

• Schuyler 1903 control groups
• McCall 1916 randomly selected groups
• McCall 1925 book on experiments
• Fisher 1936 statistical methods book

4
A Brief History of Experimental Designs

• Campbell and Stanley 1963 15 types of
  experimental designs evaluated in terms of
  threats to validity
• Cook and Campbell 1979 four types of validity
• 1980 and beyond computer enhancements to
  experimental design2

5
Key Characteristics of Experimental Designs

• Participants selected and assigned to groups
• control
• experimental
• An intervention is applied to one or more groups
• Outcomes are measured at the end of the
  experiment

6
Key Characteristics of Experimental Designs

• Procedures are designed that address potential
  threats to validity
• Internal
• External
• Construct
• Statistical Conclusion
• Statistical comparisons of different groups are
  conducted

7
Selecting Participants and Assigning Them to
Treatments

• Decide on the experimental unit of analysis to be
  treated
• individual
• group or groups
• organization
• Randomly assign individuals to groups control for
  extraneous characteristics that might influence
  the outcome

8
Selecting Participants and Assigning Them to
Treatments

• Control for extraneous factors
• random assignment (equating groups)
• pretest/posttest
• covariates
• matching participants
• selecting homogenous samples
• using blocking variables

9
Controlling for Covariates
Independent Variable
No Covariates
Dependent Variable
Covariate Introduced
Independent Variable Type of Instruction
Dependent Variable Rates of Smoking
Covariate Parents Who Smoke
Variance Removed
Variance
10
Matching Process Based on Gender
John Jim James Josh Jackson Jane Johanna Julie Jea
n Jeb
Experimental Group
Control Group
11
Applying an Intervention or Treatment

• Identify a treatment variable
• identify the conditions or levels of the variable
• Manipulate the treatment conditions

12
The Experimental Manipulation of a Treatment Group

• Independent Variables
• 1. Age (cant manipulate)
• 2. Gender (cant manipulate)
• 3. Types of Instruction (can manipulate)
• a. Lecture (control)
• b. Lecture Hazard Instruction (Comparison)
• c. Lecture Hazard Instruction slides of
  damaged lungs (experiment)

• Dependent Variable
• Frequency of Smoking

13
Threats to Statistical Conclusion Validity

• Low statistical power due to low sample size
• Violation of assumptions of statistical tests
• Use of unreliable measures

14
Threats to Internal Validity

• History
• Maturation
• Regression
• Selection
• Mortality
• Interactions with selection

15
Threats to Internal Validity

• Diffusion of treatments
• Compensatory equalization
• Compensation rivalry
• Resentful demoralization
• Testing
• Instrumentation

16
Threats to Construct Validity

• Lack of good operational definitions
• Apprehensiveness by participants
• Participants guessing what the researcher hopes
  to find

17
Threats to External Validity

• Interaction of selection and treatment
• Interaction of setting and treatment
• interaction of history and treatment

18
Treatment Comparisons in an Experiment
Phase 1 Relationship Picture

Error Correction Treatment
Spelling Accuracy
Phase 2 Timeline Picture
Test 1 Test 2 Test 3 6 Weeks 6
Weeks 6 Weeks 6 Weeks 6 Weeks 6 Weeks
Class A Regular Spelling Practice
(Control) Class B Reduced word list (Comparison)
Error Correction Treatment 6 Weeks 6 Weeks
6 Weeks
Class C Error Correction (Experimental)
Phase 3 Statistical Comparisons
Class A Class B Class C F-value
10.3 (3.6)
10.8 (4.3)
9.9 (3.9)
Test 1 Test 2 Test 3
0.27
10.6 (3.8)
13.9 (4.2)
10.7 (3.3)
4.90

plt.05
11.1 (3.3)
10.3 (3.6)
13.1 (3.8)
3.31
19
Types of Experimental Designs Between Groups

• True Experiments
• Pre- and posttest
• Posttest Only
• Quasi Experiments
• Pre- and posttest
• Posttest Only
• Factorial Designs

20
Types of Experimental Designs Within-Group or
Individual

• Time series experiments
• interrupted
• uninterrupted
• Repeated measures experiments
• Single-subject experiments
• A/B design
• Multiple baseline design
• Alternating treatments

21
Means and Main Effects of Eight groups in
Factorial Design
Depression
Medium
High
Low
Mean rate of smoking
Mean rate of smoking
Mean rate of smoking
Health lecture
Main Effects of Type of Instruction
Type of Instruction
Mean rate of smoking
Mean rate of smoking
Mean rate of smoking
Standard lecture
Main Effects of Depression
22
Graphs Showing Main and Interaction Effects
Standard lecture
(a) No interaction Effects (Parallel)
Health lecture
Extent of Smoking
(b) Interaction Effects (Crossed)
(a) Interaction Effects (Crossed)
Standard lecture
Extent of Smoking
Health lecture
(c) Interaction Effects (Not Parallel)
Standard lecture
Extent of Smoking
Health lecture
23
Steps in Conducting Experimental Research

• Decide if an experiment addresses the research
  problem
• Form hypotheses to test cause-effect
  relationships
• Select an experimental treatment and introduce it
• Identify study participants

24
Steps in Conducting Experimental Research

• Choose a type of experimental design
• Conduct the experiment
• Organize and analyze the data
• Develop an experimental research report

25
Criteria for Evaluating Experimental Research

• Does the experiment have a powerful intervention?
• Does it employ few treatment groups (e.g. only
  two)?
• Will participant profit from the intervention?
• Is there a systematic way the researcher derived
  the number of participants per group?

26
Criteria for Evaluating Experimental Research

• Were there an adequate number of participants
  used in the study?
• Were valid, reliable, and sensitive measures or
  observations used?
• Did the study control for extraneous factors?
• Did the researcher control for threats to
  internal validity?

27
Applying What you Have Learned An Experimental
Study

• Review the article and look for the following
• The research problem and use of quantitative
  research
• Use of the literature
• The purpose statement and research hypothesis
• Types and procedures of data collection
• Types and procedures of data analysis and
  interpretation
• The overall report structure