Construct and External Validity in Experimental Research ?

1
Construct and External Validity in Experimental
Research

?
Chapter 8
?Back to Brief Contents

• Construct Validity ?
• External Validity ?
• Cautions in Evaluating the External Validity of
  Experiments ?
• Relationship between Internal and External
  Validity ?

2
8.1 Construct Validity -1
?Back to Chapter Contents

• Definitionextent to which we can
• infer higher-order constructs for our
  operations
• Fig 8.1
• Constructs are used for
• Research participants
• Independent variable
• Dependent variable
• Experimental setting

3
8.1 Construct Validity -2
?Back to Chapter Contents

• Assessing Construct Validity ?
• Threats to Construct Validity ?
• Reactivity to the Experimental Situation ?
• Experimenter Effect ?

4
Assessing Construct Validity -1
?Back to Chapter Contents

• Have a clear definition of the construct of
  interest
• Problem is identifying prototypical features of
  the constructs
• Affects ability to identify the concrete
  operations used to represent the construct

5
Assessing Construct Validity -2(end)
?Back to Chapter Contents

• Assess the match between the constructs and the
  operations used to represent them
• Ways to assess
• Content validity
• Criterion-related validity
• Predictive
• concurrent
• Any other source of evidence

6
Threats to Construct Validity
?Back to Chapter Contents

• Inadequate explanation of the construct Tab
  8.1
• Construct confounding
• Mono-operation bias
• Mono-method bias
• Confounding constructs with level of constructs
• Treatment-sensitive factorial structure
• Reactive self-report changes
• Reactivity to the experimental situation
• Experimenter effects
• Novelty and disruption effects
• Compensatory equalization
• Compensatory rivalry
• Treatment diffusion

7
Threats to Construct Validity
?Back to Chapter Contents

• Inadequate explanation of the construct
• If a construct is not adequately explained and
  analyzed,
• it can lead to a set of operations that do not
  represent the construct adequately
• Construct confounding
• the operations used in a study represent more
  than one construct
• Mono-operation bias
• a study uses only one operationalization of a
  construct
• This typically results in an underrepresentation
  of the construct and lowers construct validity

8
Threats to Construct Validity
?Back to Chapter Contents

• Mono-method bias
• a study uses only one method (e.g.,
  physiological recording) to operationalize a
  construct
• The method used may influence the results
• Confounding constructs with level of constructs
• a study investigates only a few levels of a
  construct (e.g., three doses of a drug), but
  makes inferences about the overall construct
  (e.g., the overall effect of the dose)
• Treatment-sensitive factorial structure
• an instrumentation change that occurs because of
  the experimental treatment

9
Threats to Construct Validity
?Back to Chapter Contents

• Reactive self-report changes
• changes that a research participant may make on
  self-report measures as a result of a
  motivational shift after being included in the
  experimental study
• Reactivity to the experimental situation
• research participants perceptions and motives
  can affect the responses they make to the
  dependent variable
• and these responses can be interpreted as part
  of the treatment construct being tested

10
Threats to Construct Validity
?Back to Chapter Contents

• Experimenter effects
• the experimenters attributes and expectancies
  can influence the responses made by the research
  participants
• and these responses can be interpreted as part
  of the treatment construct being tested
• Novelty and disruption effects
• Research participants usually respond better to
  a new and novel situation and poorly to one that
  disrupts their routine
• These effects are part of the overall treatment
  effect

11
Threats to Construct Validity
?Back to Chapter Contents

• Compensatory equalization
• individuals try to provide the same benefits or
  services to the control group that are received
  by the experimental group
• Compensatory rivalry
• individuals resent being assigned to the control
  group and respond more negatively than would be
  expected, because of the resentment they feel
• Treatment diffusion
• individuals in one treatment group receive some
  or all of another groups treatment

12
Reactivity to the Experimental Situation 1/3

?Back to Chapter Contents

• Refers to research participants motives and
  perceptions influencing the response to the DV
• Participant Effect
• Demand characteristics
• Any of the cues available in an exp, such as
• instructions, experimenter, rumors, experimental
  settings
• Primary motive positive self-presentation

13
Reactivity to the Experimental Situation 2/3
?Back to Chapter Contents

• Conditions producing a positive self-presentation
  motive
• Tedeschi, Schlenker, Bonoma (1971)
• P believe that others view their behavior as
  indicative of their true intentions, beliefs, or
  feelings
• Exp. constructed ? P believe that others think
  their behavior is externally determined

14
Reactivity to the Experimental Situation 3/3
?Back to Chapter Contents

• Implication for research
• Intertreatment interaction
• Perception by P in different treatment groups
  that they can fulfill the positive
  self-presentation motive by responding in
  different ways
• Intratreatment interaction
• Perception by P in the same treatment group that
  they can fulfill the positive self-presentation
  motive by responding in different ways

15
Experimenter Effect 1/3
?Back to Chapter Contents

• Experimenter Effect Fig 8.2
• Experimenter has motive of supporting the study
  hypothesis
• Can unintentionally lead to recording errors

16
Experimenter Effect 2/3
?Back to Chapter Contents

• Ways experimenter may bias the study
• Experimenter attributes
• Biosocial attributes
• (e.g.) age, sex, race
• Psychosocial attributes
• psychometrically determined characteristics
• (e.g.) anxiety level, hostility,
  authoritarianism, intelligence, dominance, warmth
• Situational factors
• (e.g.) prior contact of P, naive or experienced
• Experimenter expectancies (cont.)

17
Experimenter Effect 3/3
?Back to Chapter Contents

• Ways experimenter may bias the study
• Experimenter attributes
• Experimenter expectancies
• Effect on experimenter
• (e.g.) recording errors (biased, but low error
  rate)
• Effect on research participant Tab 8.2
• Mediation of expectancies
• Handling in animal research
• Nonverbal communication in human studies
• Magnitude of expectanciescan exist in animal and
  human research and can be greater than the IV
  Fig 8.3

18
8.2 External Validity 1/4
?Back to Chapter Contents

• Definitiongeneralizing across people, settings,
  treatment variations, outcomes and times
• Represents a test of interactions
• Threats to external validity
• Population validity
• Ecological validity
• Temporal validity
• Treatment variation validity
• Outcome validity

19
8.2 External Validity 2/4
?Back to Chapter Contents

• Population Validity
• The extent to which the results of a study can
  be generalized to the larger population
• Two-step inferential process Fig 8.4 Tab 8.3
• Sample ? Experimentally accessible population
• ? Target population
• College students as P
• less emotionally and impulsively in laboratory
  studies
• Selection by Treatment Interaction
• if interaction exists
• ? cannot generalize to the
  target population

20
8.2 External Validity 3/4
?Back to Chapter Contents

• Ecological Validity
• The extent to which the results of a study can
  be generalized across settings or environmental
  conditions
• Temporal Validity
• The extent to which the results of an experiment
  can be generalized across time
• Seasonal variation
• A variation that occurs at regular time
  intervals
• Cyclical variation
• A regular variation that occurs within people
  and other organisms
• (ex) circadian rhythm, MC

21
8.2 External Validity 4/4
?Back to Chapter Contents

• Treatment Variation Validity (IV)
• The generalizability of results across variation
  of the treatment
• Outcome Validity (DV)
• The generalizability of results across different
  but related dependent variables

22
8.3 Cautions in Evaluating the External
Validity of Experiments
?Back to Chapter Contents

• Some studies conducted to increase knowledge and
  not to generalize to a real-life situation
• Some studies assess a theoretical process
• Moving out of the laboratory does not insure
  generalization

23
8.4 Relationship between Internal and
External Validity
?Back to Chapter Contents

• Relationship between internal and external
  validityoften an inverse relationship