Threats to Validity

1
Threats to Validity

• Threats to validity definition
• Theory vs. measurement
• Types of validity
• Conclusion validity
• Internal validity
• Construct validity
• External validity
• Particular threats to validity

2
Threats to validity - definition

• Researchers do not simply report facts.
• The researchers task is like describing an
  object covered by a thick blanket.

• The description is not like a photograph it is
  not veridical. It can be wrong in a variety of
  ways.
• Problems that cause our descriptions to be wrong
  are problems of validity.

3
Theory vs. measurement

• Our interest is always in the validity of a
  conclusion or inference
• That validity can be threatened either by poor
  measurement or by poor theory

Theory
Measurement
4
Theory vs. measurement

• Our theory predicts that we will observe a
  relationship between measurable variables.

• When we look, we wont see the expected
  relationship if
• the theory is incorrect, or
• the operationalizations are flawed.

5
Cause
Effect
Independent Variable
Dependent Variable
6
Theory vs. measurement

• Suppose you believe that absence makes the heart
  grow fonder.

• Your theory has two constructs
• Absence
• Fondness.
• Both must be operationalized.

7
We might not see the expected relationship
because our operationalizations are poor.
We might not see the expected relationship
because the theory is wrong
8
Types of Validity

• Our first question concerns what is going on in
  our study
• Do the data show the relationship between
  variables that our theory claims should be there?
• If so, our study has conclusion validity

9
Types of Validity

• Our second question also concerns what is going
  on in our study
• Is the relationship between the variables causal?
  (A ? B)
• If so, our study has internal validity

10
If there is some confounding variable one that
varies with our I.V. then our study lacks
internal validity
days apart
phone calls
money left
11
Types of Validity

• Our third question concerns how well our measured
  variables reflect our theoretical constructs.
• Does I.V. capture our theoretical cause?
• Does D.V. capture our theoretical effect?
• If so, we have construct validity

12
Absence
Fondness
Distance
thoughts about loved one
A different operationalization might produce
different conclusions e.g, if distance varies
with the purpose of the trip, which in turn
determines how busy the traveler is through the
day.
13
Types of Validity

• Our fourth question concerns whether our theory
  is true in the wider world.
• Will the findings in our study be replicated in
  other settings?
• If so, our conclusion has external validity
• May come at cost to internal validity

14
Particular threats to validity

• On previous slides, weve discussed several types
  of validity.
• The next slides list some threats to each type.

• You should be able to discuss and give examples
  of each of these threats.

15
Threats to Internal Validity

• History

• changes happening to only one of two or more
  groups that are compared

16
Threats to Internal Validity

• History
• Maturation

• changes in performance associated with development

17
Threats to Internal Validity

• History
• Maturation
• Test practice

• changes in performance due to experience with the
  test format

18
Threats to Internal Validity

• History
• Maturation
• Test practice
• Regression to the mean

• extreme values of measurements are unlikely to be
  repeated

19
Threats to Internal Validity

• History
• Maturation
• Test practice
• Regression to the mean
• Selection

• between-group differences that are present at the
  start

20
Threats to Internal Validity

• History
• Maturation
• Test practice
• Regression to the mean
• Selection
• Mortality

• differences produced by different rates of
  drop-out between groups

21
Threats to Conclusion validity

• increase number of items used in the task
• or use more precise measuring instruments

• Conclusion validity may be low because of
• poor reliability

22
Threats to Conclusion validity

• increase strength by increasing dose
• decrease noise by increasing reliability or
  decreasing distractions in the environment

• Conclusion validity may be low because of
• poor reliability
• weak relationship

23
Threats to Conclusion validity

• Increase number of observations, by
• increasing number of subjects
• increasing number of trials per subject

• Conclusion validity may be low because of
• poor reliability
• weak relationship
• lack of statistical power

24
Threats to Construct Validity

• Loose connection between theory and method

• Ambiguous effect of independent variables
• Hawthorne effect
• Good subject tendency
• Evaluation apprehension (social desirability,
  white coat syndrome)

25
Threats to Construct Validity

• Interaction of different treatments
• Interaction of testing and treatment.

26
Threats to External Validity

• Any dimension on which studies can differ might
  produce a failure to replicate.
• Other people

• Our subjects might be special in some way

27
Threats to External Validity

• Any dimension on which people can differ might
  produce a failure to replicate.
• Other people
• Other places

• our environment might be special in some way
• especially if we work in a lab.

28
Threats to External Validity

• Any dimension on which people can differ might
  produce a failure to replicate.
• Other people
• Other places
• Other times

• our time might be special in some way
• e.g., right after a government report on smoking
  is released, you survey smokers

29
Challenge

• For each of the following maxims, suggest some
  operationalization of the relevant constructs

• Too many cooks spoil the broth
• Many hands make light work
• You snooze, you lose
• A rolling stone gathers no moss