A TheoryGuided Interviewer Training Protocol for Survey Participation

1
A Theory-Guided Interviewer Training Protocol for
Survey Participation

• Robert M. Groves, University of Michigan and
  Joint Program in Survey Methodology
• Kate McGonagle, University of Michigan

2
Overview of Presentation

• Brief review of theoretical perspective
• Description of results of two experimental trials
• Description of training protocol
• Next steps

3
Two Very Different Processes that Produce
Nonresponse

• Noncontact of sample households
• Refusal to cooperate with the survey request

4
The Contact Process has a Small Set of Causes
Access Impediments
Social Environmental Attributes
Accessible at-home patterns
Likelihood of Contact
Socio-demographic Attributes
X
Timing of Calls
No. of Calls
5
Interviewer Behavior Reducing Noncontacts

• Access impediments
• Call on such cases early to obtain knowledge of
  source
• Evidence of general ability to overcome
  impediment
• Callbacks
• Advantage of evening, weekend
• Average 2-3 calls to first contact

6
Cooperation is a Complex Phenomenon
Social Environment
Design Features
Household/ Person Predispositions
Interviewer Attributes/ Behavior
Householder-Interviewer Interaction
Decision to Participate
7
Three Theoretical Concepts

• Leverage-Salience
• Tailoring
• Maintaining interaction

8
Leverage-Salience Theory of Survey Participation
Authority of Sponsor
Authority of Sponsor
Incentive
Topic
Incentive
Burden
Topic
9
Effects of Salience Weights on Propensity to
Cooperate
Authority of Sponsor
Burden
Authority of Sponsor
Incentive
Incentive
Topic
Topic
10
Tailoring

• I give the introduction and listen to what
  they say. I then respond to them on an
  individual basis, according to their response.
  Almost all responses are a little different, and
  you need an ability to intuitively understand
  what they are saying.

11
Lessons for Interviewer Behavior

• Scripting of interviewers reduces likelihood that
  they address concerns of different sample persons
• Interviewer training must train them to
• listen for concerns
• identify a way to address them
• deliver answer quickly, in native language

12
Goals of a Training Protocol

• Use expert knowledge among best interviewers
• Identify common concerns as expressed in native
  language of target population
• Train in diagnosis, customized reply, and
  delivery speed
• Evaluate absorption of training

13
Experiment 1

• Current Employment Statistics program of the U.S.
  Bureau of Labor Statistics, asks each sample
  employer to report six items concerning the
  counts and payroll for various types of employees
• Request protocol involves three steps
• a sample refinement step
• an advance mailing package sent to the contact
  person,
• an enrollment call

14
Experiment 1 Research Design

• Comparisons of interviewers performances before
  and after training workshop
• 320 cases from stratified samples of employers in
  4 states were contacted in Phase 1 (pre-training)
  and 329 in Phase 2 (post-training)
• 17 interviewers in 2-day training workshop

15
Pre- and Post-Training Cooperation Rates for
Experiment 1
16
Random Coefficient Models Predicting Likelihood
of Cooperation in Experiment 1
lnpij/(1 pij) ? 0j ? 1j (POSTij1) ? ?
k (SIZEk 1) ?ij
Two models compared for the random coefficients
1) Base Model ? 0j ?00 u0j ? 1j ?
10 u1j 2) Learning Model ? 0j ? 00
u0j ? 1j ? 10 ? 11 (GRADE)j u1j
17
Values of Estimated Coefficients in Experiment 1
Two models compared for the random coefficients
1) Base Model ? 1j 0.43 u1j
(0.21) 2) Learning Model ? 1j
0.40 0.07 (GRADE)j u1j (0.46) (0.36)
18
Conclusions from Experiment 1

• Apparent positive training effect (10 percentage
  points)
• Reduced interviewer variation in cooperation
  rates
• Potential confounding of effects of experience
  and effects of training
• Weak predictive power of grade on evaluation

19
Design of Experiment 2

• U.S. Census of Agriculture, followup of
  nonrespondent farm operators by telephone
• Interviewing organized by state (5 experimental
  states, 5 control states)
• 96 interviewers given experimental training
  assigned 11,000 cases
• 99 interviewers assigned 12,500 cases in control
  states

20
Pre-Post Cooperation Rate Differences by
Treatment Group, Experiment 2
Control
Experimental
21
Pre- and Post-Training Cooperation Rates by
Experiment 2 Group
22
Pre-Post Interviewer Cooperation Rates by
Treatment Group
Pre
Post
Pre
Post
Experimental
Control
23
Random Coefficient Models Predicting Likelihood
of Cooperation in Experiment 2
lnpij/(1 pij) ?0j ?1j (POSTij1) ? 2
(FORMij 1) ? ij
Two models compared for the random coefficients
1) Training Model ? 0j ? 00 u0j ?
1j ? 10 ? 10 (TRAINING1)j u1j 2)
Learning Model ? 0j ? 00 u0j ? 1j ? 10
? 10 (TRAINING1)j ? 12 (GRADE)j u1j
24
Estimated Coefficients for Experiment 2
Two models compared for the random coefficients
1) Base Model ? 1j 0.038 0.63
(TRAINING1)j u1j (0.08)
(0.10) 2) Learning Model ? 1j 0.038 -
0.15 (TRAINING1)j 0.051 (GRADE)j
(0.08) (0.51) (0.033) u1j
25
Conclusions from Experiment 2

• Training produces positive effects beyond those
  expected from experience
• Interviewer variation in performance reduced by
  improving the low performers
• Evaluation tool still needs development

26
What are the Steps of the Training Protocol?

• Focus groups of experienced interviewers,
  reporting words used to communicate concerns
  about the survey request (200-400 examples
  identified)
• Thematic sets were refined through feedback from
  senior interviewers (9 main thematic groups
  identified)

27
Themes of Sample Person Concerns

• Time Its harvest time
• Burden I want to be paid for my time
• Government Surveys drive the prices down
• Privacy Thats none of your business
• Crisis/Change Im bankrupt. I lost the farm
• Small Operator Go after the big guy
• Policy We dont do surveys anymore
• Burn-out Im bowing out this time
• Pass-off The countys got that stuff

28
Steps of the Training Regimen Development (contd.)

• The training workshop in learning to
• understand the themes of concerns
• classify actual wording into those themes
• deliver, in compatible wording, a set of
  statements relevant to the concerns
• increase the speed of performance on the above
• Structured evaluation of how well interviewers
  obtained the skills taught by the workshop

29
Classification of Actual Wording into Themes

• Interviewers completed a match-column exercise to
  learn to link theme groupings to response
  verbatims
• Pairs of interviewers role-play
• Listened to unknown concerns
• Articulated an oral diagnosis giving thematic
  concern type (Diagnosis)
• Selected and delivered a relevant response

30
Desirable Behaviors to Address Concerns

• Lecture and discussion on the importance of call
  preparation and initial contact protocols
• Learning the nature of previous reluctance
• Anticipating respondent concerns
• Sensitivity to respondent tone and nonverbal cues
• Listening exercise using a pre-recorded series of
  study-specific introductions
• Group critique of content and delivery of
  introductions
• Suggested improvements brainstormed

31
Delivery of Statements Relevant to Respondent
Concerns

• Paired exercise Interviewers used scripted
  interactions to practice responding to one them
  (government concerns)
• Triad exercise Interviewers divided into groups
  of 3 with rotating roles
• Silent diagnosis of wide variety respondent
  concerns
• Delivery of rebuttal
• Structured written critique and oral delivery

32
Increasing the Speed of Performance

• Rapid Fire Exercise 1. Divided interviewers
  into small groups of 6-8 members. Exercises
  focused on
• Interviewers crafting rebuttals to concerns
  delivered by facilitator
• Eliciting suggestions and comments, giving
  alternative approaches from group members
• Rapid Fire Exercise 2. Focus on increasing the3
  speed of performance by
• Elimination of comments between interactions
• Facilitator increasing the pace of role-play

33
Structured Evaluation of Interviewer Skills

• Written evaluation Interviewers are given 5
  respondent concerns
• Listen to concern delivered orally by facilitator
• Identify theme group for respondent concern
• Select a relevant rebuttal for the concern
• Quickly draft a handwritten rebuttal addressing
  the concern
• Complete handwritten response to each respondent
  concern within a time limit of 2 minutes

34
Summary

• On multiple surveys, training leads to higher
  cooperation
• Nature of training effects are to bring up the
  low performers to higher levels
• We have not yet produced a trainee-level
  evaluation instrument with predictive value

35
Work in Process

• Development of new post-training assessment
• Telephone call-in by trainee to scripted staff
  member
• Simulated survey introduction
• Successive concerns raised by staff member
• Grading of interviewer behavior