Validity: Introduction

1
Validity Introduction
2
Reliability and Validity
Reliability Low
High








Validity Low








High








3
Types of Error in Repeated Measurements

True value

4
The Validation Sequence

• Does it measure what its supposed to measure?
• First, define what you want to measure
  (conceptual basis)
• Select indicators or items that represent that
  topic (content validity a sampling issue)
• Check that items are clear, comprehensible and
  relevant (face validity, sensibility).
• This produces a pool of items ready for item
  analysis stage, which involves administering the
  test and analyzing responses.

5
Validation Sequence (2) Internal structure

• Item analysis refers to a series of checks on the
  performance of each item. Some fall under the
  heading of reliability, some validity. Faulty
  items are discarded or replaced. Analyses
  include
• Item distributions missing values an item that
  does not vary cannot measure anything
• Correlations among items, maybe with factor
  analysis
• Item response theory (IRT) analyses

6
Validation Sequence (3)External associations

• Criterion validation, if a gold standard
  exists. Sensitivity specificity are normal
  statistics.
• Correlational evidence, leading to construct
  validation, where there is no single, clear gold
  standard. Correlations are the normal statistic.
  
• Correlations often divided into convergent and
  discriminant coefficients, according to
  hypothesized associations
• These analyses tend to use the entire test
  administered to selected samples inadequate
  performance leads back to basic design

7
Validation Sequence (4)Group Discrimination

• Once you show that the test correlates with other
  measures as intended, its actual performance is
  evaluated in rating groups of respondents.
  Analyses generally use representative samples
• Known groups (can it distinguish well from
  sick similar to criterion validity)
• Sensitivity to change over time (relevant to an
  evaluative measure) responsiveness.
• Do scores show ceiling or floor effects?

8
Conclusion

• Validation is rarely complete. Many instruments
  continue to be checked for validity 20 years
  after their invention. Times change, phrasing
  makes old items obsolete, etc.
• It is long and expensive. Basic test development
  and validation may take 3 - 5 years its not a
  thesis project.
• Remember validity is about the interpretation of
  scores. It is a relative concept a test is not
  valid or invalid, but only valid or not for a
  particular application.
• The viagara principle a test intended for one
  purpose may prove good for an unanticipated
  application.