Assignment Two

1

• Assignment Two
• Issues in Measurement
• Reliability and Validity

2
Reliability A measure of consistency

• Interrater
• Correlation of ratings from two different raters
  on the same test how consistent are my scores
  with someone elses scores?
• Test-Retest
• Correlation of a test at time 1 with the same
  test at time 2 how consistent are my scores from
  two different occasions?
• Internal consistency
• Correlation of one part of a test with another
  part of the same test how consistent are my
  scores from the first half of a test with my
  scores from the second half of a test?

3
ValidityAre your measures being used the way
you intend them to be used?

• Construct
• Is your measurement tool representing the
  construct of interest?
• Statistical
• Were the analysis tools appropriately chosen?
• External
• Do the results generalize to other people,
  places, or times?
• Convergent
• Do different studies, using different operational
  definitions, produce similar results?
• Predictor (predictive)
• How well does the measure predict, or correlate
  with, another variable that it ought to predict
  or correlate with?
• E.g. If we are interested in the SAT as a valid
  measure of academic ability, one way to test this
  is to find out whether it is a valid predictor of
  other measures of academic ability. Does SAT
  predict future GPA? Does it predict or correlate
  with IQ score?

4
Reliability vs. Validity

• Reliability consistency of measurement
• Correlation of 0.80 is threshold for reliability
• (Review from psyc 30 correlation coefficients
  range from -1 to 1, with 0 no linear
  relationship between the variables)
• Predictive validity degree to which your measure
  predicts what it ought to predict
• Not a property of measurement tool depends on
  how the measure is used
• Correlation of 0.50 is good predictive validity

5
Scale Attenuation

• When scores cluster around a certain point in a
  scale, this may be indicative of a problem with
  your measurement tool.
• Lets say you would like to know if SAT is a
  valid predictor of GPA. How might your
  correlation be effected if you only select
  students with an SAT score over 1400?

6
Reliability vs. Validity
Reliable, but not Valid
7
Reliability vs. Validity
Not Reliable, not Valid
8
Reliability vs. Validity
Reliable and Valid
9
Assignment 2

• Due Tue 1/30, 800am
• Grading 6 points
• Length max 2 pages. Can answer each question
  separately each answer should be more than a
  sentence but less than a paragraph.
• Attach data on a separate page.
• Data will be generated with a partner, but all
  reports must be written and submitted
  independently.

10
Goals of assignment 2

• To learn more about defining and measuring
  psychological variables
• The variable of interest here is
  "trustworthiness. In Chapter 4 of the textbook,
  and in the lectures, several criteria are
  discussed that would apply to measurement of such
  variables, including issues of reliability,
  validity, and scale attenuation.
• These issues will be the focus of your report.
• How should trustworthiness be measured? How might
  it be related to other variables, such as
  intelligence and attractiveness?

11
Questions to Answer

• (a) You have been asked to measure
  "trustworthiness." Considering the procedure you
  used, how would you define this variable
  operationally?
• (b) What was the interrater reliability of your
  measurements of trustworthiness? Use the data
  generated by you and your partner. Would you
  consider this reliability acceptable?
• (c) How would you evaluate the scale attenuation
  of your measurements of trustworthiness? Use the
  data generated by you and your partner.
• (d) Suppose you wanted to assess the predictive
  validity of your measurements of trustworthiness.
  What would be a good way to do this? What could
  your trustworthiness ratings predict? (It is not
  necessary to assess validity for this assignment
  just suggest how it might be done.)
• (e) Examine the data for judgments of
  intelligence and attractiveness. Do these data
  suggest any theoretical basis for the judgments
  of trustworthiness?
• (f) Do you think the procedure used in this
  assignment is a good way to measure
  trustworthiness? Why, or why not? Can you suggest
  a better way?