Part 2: Quantitative Methods

1
Part 2 Quantitative Methods

• October 9, 2006

2
Validity

• Face
• Does it appear to measure what it purports to
  measure?
• Content
• Do the items cover the domain?
• Construct
• Does it measure the unobservable attribute that
  it purports to measure?

3
Validity

• Criterion
• Predictive
• Concurrent
• Consequential

4
Types of validity (cont.)
Here the instrument samples some and only of the
construct
5
Types of validity
Here the instrument samples all and more of the
construct
6
The construct
Here the instrument fails to sample ANY of the
construct
The instrument
7
The construct
Here the instrument samples some but not all of
the construct
The instrument
8
Perfection!
9
Reliability and Validity
10
In groups of 3 to 4

• Sampling
• What is the target population?
• What sampling procedure was used?
• Do you think the sample is representative?
• Why or why not?
• Measurement
• What types of reliability and validity evidence
  are provided?
• What else would you like to know?

11
Ways to Classify Instruments

• Who Provides the Information?
• Themselves Self-report data
• Directly or indirectly from the subjects of the
  study
• From informants (people who are knowledgeable
  about the subjects and provide this information)

12
Types of Researcher-completed Instruments

• Rating scales
• Interview schedules
• Tally sheets
• Flowcharts

• Performance checklists
• Observation forms

13
Excerpt from a Behavior Rating Scale for Teachers
Instructions For each of the behaviors
listed below, circle the appropriate number,
using the following key 5 Excellent, 4
Above Average, 3 Average, 2 Below Average, 1
Poor. A. Explains course material
clearly. 1 2 3 4 5 B. Establishes rapport with
students. 1 2 3 4 5 C. Asks high-level
questions. 1 2 3 4 5 D. Varies class
activities. 1 2 3 4 5
14
Excerpt from a Graphic Rating Scale
Instructions Indicate the quality of the
students participation in the following class
activities by placing an X anywhere along each
line. Always Frequently Occasionally Seldom Neve
r 1. Listens to teachers instructions.
Always Frequently
Occasionally Seldom
Never 2. Listens to the opinions of other
students. Always
Frequently Occasionally
Seldom Never 3. Offers own opinions
in class discussions.
15
Sample Observation Form
16
Discussion Analysis Tally Sheet
17
Performance Checklist Noting Student Actions
18
Types of Subject-completed Instruments

• Questionnaires
• Self-checklists
• Attitude scales
• Personality inventories

• Achievement/aptitude tests
• Performance tests
• Projective devices

19
Example of a Self-Checklist
20
Example of Items from a Likert Scale
21
Example of the Semantic Differential
22
Pictorial Attitude Scale for Use with Young
Children
23
Sample Items from a Personality Inventory
24
Sample Items from an Achievement Test
25
Sample Item from an Aptitude Test
26
Sample Items from an Intelligence Test
27
Item Formats

• Questions used in a subject-completed instrument
  can take many forms but are classified as either
  selection or supply items.
• Examples of selection items are
• True-false items
• Matching items
• Multiple choice items
• Interpretive exercises
• Examples of supply items are
• Short answer items
• Essay questions

28
Norm-Referenced vs. Criterion-Referenced
Instruments

• All derived scores give meaning to individual
  scores by comparing them to the scores of a
  group.
• The group used to determine derived scores is
  called the norm group and the instruments that
  provide such scores are referred to as
  norm-referenced instruments.
• An alternative to the use of achievement or
  performance instruments is to use a
  criterion-referenced test.
• This is based on a specific goal or target
  (criterion) for each learner to achieve.
• The difference between the two tests is that the
  criterion referenced tests focus more directly on
  instruction.

29
Experimental Research
30
The (Never-Ending) Search for Causation

• Establishing causation among variables
• Produces increased understanding of those
  variables
• Results in the ability to manipulate conditions
  in order to produce desired changes

31
Experimental Research

• Can demonstrate cause-and-effect very
  convincingly
• Very stringent research design requirements
• Experimental design requires
• Random assignment to groups (experimental and
  control)
• Independent treatment variable that can be
  applied to the experimental group
• Dependent variable that can be measured in all
  groups

32
Quasi-Experimental Research

• Used in place of experimental research when
  random assignment to groups is not feasible
• Otherwise, very similar to true experimental
  research

33
Fundamentals of Experimental and
Quasi-Experimental Research

• Cause and effect
• Incorporates a temporal elementthe cause is a
  condition that exists prior to the effect effect
  is a condition that occurs after the cause
• There exists a logical connectioncause-and-effe
  ct is demonstrated when manipulation of the
  independent variable results in differences in
  the dependent variable (as evidenced by comparing
  the experimental group to the control group)

34
What Aids Our Causal Arguments?

• Theory
• "causes certainly are connected to effects but
  this is because our theories connect them, not
  because the world is held together by cosmic
  glue. The world may be glued together by
  imponderables, but that is irrelevant for
  understanding causal explanation." Hanson, 1958.
• Temporal Elements
• Design
• "No causation without manipulation" Rubin
  Holland

35
Fundamentals of Experimental and
Quasi-Experimental Research

• Random selection and random assignment
• Distinguish between selection and assignment
• Random selection helps to assure population
  validity
• If you incorporate random assignment

Experimental research

• If you do not use random assignment

Quasi-experimental research
36
Fundamentals of Experimental and
Quasi-Experimental Research (contd.)

• When to use experimental research design
• If you strongly suspect a cause-and-effect
  relationship exists between two conditions, and
• The independent variable can be introduced to
  participants and can be manipulated, and
• The resulting dependent variable can be measured
  for all participants

37
Internal and External Validity

• Validity of research refers to the degree to
  which the conclusions are accurate and
  generalizable
• Both experimental and quasi-experimental research
  are subject to threats to validity
• If threats are not controlled for, they may
  introduce error into the study, which will lead
  to misleading conclusions

38
Threats to External Validity

• External validityextent to which the results can
  be generalized to other groups or settings
• Population validitydegree of similarity among
  sample used, population from which it came, and
  target population
• Ecological validityphysical or emotional
  situation or setting that may have been unique to
  the experiment
• If the treatment effects can be obtained only
  under a limited set of conditions or only by the
  original researcher the findings have low
  ecological validity.

39
Threats to External Validity

• Selection bias
• if sample is biased you cannot generalize to the
  population.
• Reactive effects
• Experimental setting - differs from natural
  setting.
• Testing pretest influences how subjects respond
  to the treatment.
• Multiple-treatment inference
• If the subjects are exposed to more than one
  treatment, then the findings could only be
  generalized to individuals exposed to the same
  treatments in the same order of presentation.

40
Threats to Internal Validity

• Internal validityextent to which differences on
  the dependent variable are a direct result of the
  manipulation of the independent variable
• Historywhen factors other than treatment can
  exert influence over the results problematic
  over time
• Maturationwhen changes occur in dependent
  variable that may be due to natural developmental
  changes problematic over time
• Testingpretest may give clues to treatment or
  posttest and may result in improved posttest
  scores
• Instrumentation Nature of outcome measure has
  changed.

41
Threats to Internal Validity (contd.)

• Regression Tendency of extreme scores to be
  nearer to the mean at retest
• Differential selection of participantsparticipant
  s are not selected/assigned randomly
• Attrition (mortality)loss of participants
• Experimental treatment diffusion Control
  conditions receive experimental treatment.

42
Experimental and Quasi-Experimental Research
Designs

• Commonly used experimental design notation
• X1 treatment group
• X2 control/comparison group
• O observation (pretest, posttest, etc.)
• R random assignment

43
Common Experimental Designs

• Single-group pretest-treatment-posttest design

O X O

• Technically, a pre-experimental design (only one
  group therefore, no random assignment exists)
• Overall, a weak design
• Why?

44
Common Experimental Designs (contd.)

• Two-group treatment-posttest-only design

R X1 O R X2 O

• Here, we have random assignment to experimental,
  control groups
• A better design, but still weakcannot be sure
  that groups were equivalent to begin with

45
Common Experimental Designs (contd.)

• Two-group pretest-treatment-posttest design

R O X1 O R O X2 O

• A substantially improved designpreviously
  identified errors have been reduced

46
Common Experimental Designs (contd.)

• Solomon four-group design

R O X1 O R O X2 O R X1 O R X2 O

• A much improved designhow??
• One serious drawbackrequires twice as many
  participants

47
Common Experimental Designs (contd.)

• Factorial designs

R O X1 g1 O R O X2 g1 O R O X1 g2
O R O X2 g2 O

• Incorporates two or more factors
• Enables researcher to detect differential
  differences (effects apparent only on certain
  combinations of levels of independent variables)

48
Common Experimental Designs (contd.)

• Single-participant measurement-treatment-measureme
  nt designs

O O O X O X O O
O O

• Purpose is to monitor effects on one subject
• Results can be generalized only with great caution

49
Common Quasi-Experimental Designs

• Posttest-only design with nonequivalent groups

X1 O X2 O

• Uses two groups from same population
• Questions must be addressed regarding equivalency
  of groups prior to introduction of treatment

50
Common Quasi-Experimental Designs (contd.)

• Pretest-posttest design with nonequivalent groups

O X1 O O X2 O

• A stronger designpretest may be used to
  establish group equivalency

51
Similarities Between Experimental and
Quasi-Experimental Research

• Cause-and-effect relationship is hypothesized
• Participants are randomly assigned (experimental)
  or nonrandomly assigned (quasi-experimental)
• Application of an experimental treatment by
  researcher
• Following the treatment, all participants are
  measured on the dependent variable
• Data are usually quantitative and analyzed by
  looking for significant differences on the
  dependent variable