Content Analysis: Measurement and Validity

1
Content AnalysisMeasurement and Validity

• Kimberly A. Neuendorf, Ph.D.
• Cleveland State University
• Fall 2011

2
Measurement

• Assignment of numerals to objects or events
  according to rules (Stevens, 1951)

3
Operationalization (Measurement)

• Categories that are exhaustive and mutually
  exclusive
• High and appropriate level of measurement
• Ratio
• Interval
• Ordinal
• Nominal

4
Evaluating MeasuresComparing Reliability,
Accuracy (lack of Bias), and Precision

• Targetsalso see p. 114 of book.
• See also
• Carmines, E. G., Zeller, R. A. (1979).
  Reliability and validity assessment. Newbury
  Park, CA Sage Publications.

5
(No Transcript)
6
High Reliability High Accuracy High Precision
7
(No Transcript)
8
High Reliability Low Accuracy High Precision
9
(No Transcript)
10
Low Reliability Moderate Accuracy High Precision
11
(No Transcript)
12
Reliability? High Accuracy Low Precision
13
Validity

• Internal Validity
• Conceptual definition matches
  Operationalization
• The standard of having a good measureAre we
  hitting the target we want?
• External Validity
• Generalizability
• Sample
• Ecological validity
• Replicability

14
Validity

• Face Validity
• Criterion Validity
• Relate to established standard or behavior
• Concurrent
• Predictive
• e.g., Oliver Kalyanaraman (2002) study of
  sexual and violent content in trailers and films.
  . . Criterion is the MPAA rating

15
Validity

• Content Validity

Construct ???
16
Validity

• Content Validity

17
Validity

• Content Validity

18
Validity

• Content Validity

High internal consistency reliability. . .
19
Validity

• Construct Validity
• Related to other measures (constructs) in ways
  consistent with theory
• Convergentrelationships as expected
• Discriminantlack of relationships, as expected

20
Construct Validity

• Smith volume (1992) on thematic content
  analysispsychographic applicationshas examples
• Trust-mistrust coding system for TAT stories
  construct validated against
• Other thematic coding systems (e.g., Affiliation
  Intimacy)
• Demographics (e.g., Mothers feeding habits
  during infancy (more or less principled),
  moderated by gender age)
• Self-reported behaviors (e.g., Reported positive
  relationships with others Divorce, family fights
  in last year Drug use)
• Personality measures (e.g., Impulsivity Locus of
  Control)

21
Validity Assessment CATA

• Short et al. (2010) provides a model for validity
  and CATA, using tests of a new dictionary for
  entrepreneurial orientation
• Content Validity
• Deductive
• Inductive
• External Validity
• Reliability
• Dimensionality (a type of Construct
  Validity--Discriminant)
• Predictive Validity (seems to really be Construct
  ValidityConvergent???)

22
Validity Assessment CATA

• A Slight Adaptation of Short et al. (2010),
  converging with Carmines and Zeller (1979)
• Face Validity
• Content Validity
• Deductive
• Inductive
• External Validity
• Construct Validity--Dimensionality (Discriminant
  Validity)
• Construct Validity--Predictive (Convergent
  Validity)
• Criterion Validity (Concurrent or Predictive)
• Reliabilityseparate, but without it, validity
  is also compromised

23
Apply to the iCarly Case? (Disrespectful
Behaviors)

• A Slight Adaptation of Short et al. (2010),
  converging with Carmines and Zeller (1979)
• Face Validity
• Content Validity
• Deductive
• Inductive
• External Validity
• Construct Validity--Dimensionality (Discriminant
  Validity)
• Construct Validity--Predictive (Convergent
  Validity)
• Criterion Validity (Concurrent or Predictive)
• Reliabilityseparate, but without it, validity
  is also compromised

24

• pause