L4 - Assessment Methods (II)

1
L4 - Assessment Methods (II)

• SOC325 Behavior at work
• Ref Spector, Ch. 2 (p.32-36)

2

• Outline
• 1. Different types of assessment methods for
  personnel selection
• Interviews
• Biographical information
• Work sample
• Psychological tests
• Assessment center
• 2. Assessing test reliability and validity

3

• To have an accurate assessment of eye-sight, we
  need a good optician
• How do you pick one?
• How to pick a good assessment test?
• What it takes to be a GOOD test?

4
A scientific way to evaluate a test

• A good test should demonstrate
• High reliability
• Evidence for its validity

5
Reliability

• A test is reliable when it gives consistent
  measurements across repeated observations.

NO! Im 168 lb only!
He read
Hey! Its not reliable!
168 lb
You read
170 lb
6
Reliability

• Test-retest reliability A test has high
  test-retest reliability when it gives consistent
  measurement across times.

160 lb
160 lb
160 lb
1 min. later
1 min. later
7
Reliability

• Inter-rater reliability A test has high
  inter-rater reliability when different users
  (raters) using it give consistent measurements of
  the same object.

8
Reliability

• Internal consistency reliability
• For tests that use multiple items to give a
  measurement
• Each item should be consistently measuring the
  same variable
• Overall, I am satisfied with the teachers
  performance.
• v.s.
• 1. The teacher has structured the presentation
  well.
• 2. The teacher encouraged students
  participation.
• 3. The size of students desk was suitable for
  writing.
• 4. The teacher was responsive to students
  questions.

9
Reliability

• Internal consistency reliability
• A test has high ICR when multiple items give
  similar or consistent measurement.
• 1. The teacher was structured the presentation
  well.
• 2. The teacher encouraged students
  participation.
• 3. The size of students desk was suitable for
  writing.
• 4. The teacher was responsive to students
  questions.

10
A good test has high reliability

• Test-retest across time (same user same
  target)
• Inter-rater across raters/users (same target
  same time)
• Internal consistency reliability across items
  (same target same user same time)

11
The concepts of validity

• Target Instrument
• Length Income
• Health Ruler
• Intelligence O-P scale
• Optimism Blood Pressure
• Social Status IQ test

12
Validity

• Is the tool appropriate to measure the KSAO that
  we are interested in?
• A test can be reliable but not valid! (Ruler
  Optimism)
• A test is valid when there is evidence that the
  test score reflects what the test is supposed to
  measure
• How do we know whether a ruler or O-P scale
  really measures optimism?
• 3 types of evidence
• Face Content Criterion-related validity

13
Validity

• Face validity the test looks like what it was
  designed to measure (subjective).
• ????, ???, ???
• ?????????.
• O-P
• ????????
• ????????????
• ?????, ???????

14
Validity

• Content validity the test adequately covers the
  content of the variable (subjective).
• ????????????
• ??????????
• ??????????
• ???????????
• ?????
• O-P
• ????????
• ????????????
• ?????, ???????

15
Validity

• Criterion-related validity
• IQ test score relates to academic performance,
  speed of learning new words
• Standford Binet IQ test Hi score more A
  grades than low score
• ?? (???) ????
• There is evidence for CR validity when the test
  score relates to another variable which is
  theoretically related (objective).

16
Validity

• Criterion-related validity Examples
• O-P
• Hi score in O-P less likely to be depressed low
  score more likely to be
• Psychological assessment scores should relate to
  job performance!
• Higher Ability tests score higher performance
  rating

17
A good test has evidence for its validity

• Face the test looks like what it intends to
  measure (Picture drawing interpersonal skills)
• Content the content of the test has adequate
  coverage of the variable (???? ????)
• Criterion-related test score relates to another
  theoretically related variable (IQ A Ability
  test score job performance)

18
Typical Criterion-related Validity Coefficients

• Age r .00
• Handwriting analysis (Graphology) .00
• Vocational interests tests .10
• College grades .13
• Reference checks .17
• Experience .18
• Personality tests .24
• Assessment centers .25
• Structured interviews .34
• Biographical information blanks .35
• Job knowledge tests .50
• Cognitive and psychomotor ability tests .53

19
How to pick a good assessment test?

• Ask for information about the tests
• Reliability
• Validity
• JA ? Task KSAOs ? Pick some reliable and valid
  assessment tests ? Now, we know how to assess the
  job applicants!
• NEXT, recruitment and selection