Qualities of Research: Validity and Reliability

1
Qualities of Research Validity and Reliability

• GU Yongqi
• Victoria University of Wellington
• New Zealand
• peter.gu_at_vuw.ac.nz

2
Validity

• Validfaithful, true
• What is assessed is indeed what is intended to be
  assessed
• What we measure is indeed what we intend to
  measure
• What we claim to have found (our interpretation
  of our findings) is indeed what we have found

3
Types of validity

• Internal validity (justifiability)
• To what extent can the claimed findings be
  justified or true?
• e.g., in an experiment Can the changes in the
  dependent variable be attributed only to the
  experimental treatment?
• Every design aspect can threaten a studys
  internal validity, e.g.,
• Subjects and subject selection, e.g.,
  pre-existing differences-info in Paper 1?
• Variable control and treatment, e.g., placebo
  effect, extra time in experimental group
• Materials and measurement, e.g., questionnaires

4
Validity of instrument 2005????????????????????

• The following test items appeared on a
  large-scale test (taken by 365,000 people on 26
  November 2005).
• Why arent there polar bears in the South Pole?
• Why do crocodiles shed tears in front of food?
• Why do we feel that the air is fresher after a
  thunder storm in summer?
• Why cant we drink sea water?

5
Types of validity

• External validity (generalisability)
• Are the findings true for other populations?
• Are the findings true for other contexts?
• Read Wen (2004), pp.130-139, p.248 for
  details

6
Reliability

• Reliability
• Comparability
• Consistency
• Trustworthiness
• The measurement is consistent and stable

7
Types of reliability

• Stability reliability (consistency over time),
  test-retest (replication study should reveal
  similar results)
• Alternate-form reliability (parallel study should
  reveal similar results)
• Internal consistency reliability (split-half
  Cronbachs coefficient alpha K-R21)
• Inter-rater/intra-rater reliability (percentage
  of agreement correlation)

8
Version 1.4 Reliability analysis
9
Deleted item 38
10
Reliability reported in Paper 1

• p. 192
• Coding agreements between two researchers/coders
  .90
• Formula
• of agreed codings x 2
• Reliability -------------------------------
• Total of codings

11
Validity and reliability

• Real validity implies reliability.
• If something is consistent (reliable), it may not
  necessarily be true (valid) but if something is
  true, it should be consistently true.

12
Evaluating Paper 1

• Are the claims of findings and interpretations
  true? (Validity)
• Why?
• Why not?
• If exactly the same materials and procedures were
  used again, would the same results be obtained?
  (Reliability)
• Why?
• Why not?