Dependent Interviewing: Evidence from Field Tests The SIPP Methods Panel

1
Dependent Interviewing Evidence from Field
TestsThe SIPP Methods Panel

• Nancy Bates and Joanne Pascale
• U.S. Census Bureau
• Seminar on Dependent Interviewing Univ. of Essex,
  September 17, 2004

2
Acknowledgements

• Pat Doyle, Jeff Moore, Joanne Pascale, Julia
  Klein Griffiths, Anna Chan
• Heather Holbert, Elaine Hock, Johanna Rupp,
  Aniekan Okon, Ceci Villa, Judy Eargle
• U.S. Census Bureau Field Division

3
Methods Panel Research Project

• 4-yr research project to study and refine
  recommendations
• Extant data and research, cognitive assessments,
  large-scale field experiments, interviewer
  assessments
• 3 Field Experiments
• Treatment groupexperimental instrument
• Control groupSIPP production instrument
• 1000 households per treatment
• Conducted in 6 Regional Offices
• CAPI interview (CASES language)

4
Methods Panel Research Project

• 3 Field Experiments
• Experiment in 2000 Wave 1
• Experiments in 2001 2002 Waves 1 and 2
• Interviewer assessments, data analysis
• Response Rates from 83 - 92
• Produced SIPP 2004 instrument

5
Cognitive Assessments

• Wave 1 34 households interviewed
• Wave 2 4 months later, 23 households
  re-interviewed
• Cognitive interview followed by debriefing
• Reactions to dependent interviewing techniques
• Sharing of data with other members
• Reaction to Respondent Identification Policy
  (RIP)

6
Cognitive Assessments

• RIP Question We re-contact households every 4
  months to update information. If we talk to
  someone else in your household next time, instead
  of you, is it OK if we use your answers as a
  starting point?
• D.I. did not elicit privacy concerns
• Respondents expected previous information to be
  stored
• Few respondents declined RIP request
  (misunderstood?)

7
Interviewer Debriefings

• Pencil and paper debriefing after each wave (Wave
  1 n152 Wave 2 n131)
• Section in Wave 2 about D.I. Techniques
• Use of Same as last time reminders
• Dependent questions vs. from scratch
• Dependent question nonresponse follow-ups
• Correcting mis-reported breaks in receipt
• Rated on 5 point scale (Strongly disagree
  Strongly agree)

8
Interviewer Debriefings

• D.I. preferred over from scratch (assets)
• D.I. perceived as easier for respondent smoother
  interview easier to administer more accurate
  (earnings, unearned income, assets, health
  insurance)
• D.I. not perceived more accurate in correcting
  seam bias (not sig. different from control).
• D.I. perceived more accurate regarding health
  insurance coverage

9
Attrition

• Response Rates (Unit-level attrition)
• Learning Curve effect (year)
• D.I. effect (treatment)

10
(No Transcript)
11
Attrition

• On surface, little evidence D.I. impacted
  attrition
• No significant difference in test/control
  comparisons
• Closer look at Wave 1 across years and treatment
  groups
• Logistic regression
• Significant (plt.10) main effect for year
  (negative)
• Borderline (p.11) yeartreatment interaction
• Nonresponse decline across years is sig. for test
  group but not control
• Wave 2 exhibits similar trend, but interaction
  not sig.

12
Dependent Interview Nonresponse Follow-up
Techniques

• If Dont know Refused or Same as last
  time.
• Earnings Things may have changed since then, but
  I have recorded from last time that NAME earned
  about X,XXX a month from this job with EMPLOYER
  NAME. Does that still sound about right?
• Unearned Income It says here that NAME
  received XXX in food stamp benefits last
  MONTH. Does that still sound about right?
• Asset Income Things may have changed a lot but I
  have recorded from last time your income from
  these rental properties was about X,XXX
  annually. Does that still sound about right?

13
Assessing Dependent Interview Nonresponse
Follow-up Techniques

• Item nonresponse for income amounts (dont know,
  refused)
• Computed aggregate nonresponse ratio
• D.K. Refused
• questions asked
• Average of ratios across all adults in universe

14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
SIPP interview months and their associated
reference period months (example)
18
First Interview
Second Interview
19
Dependent Interview Seam-Bias Reduction Techniques

• Unearned Income (and School Enrollment)
• If reported in M5 previous Last time I recorded
  that you received food stamps in month. Is
  that correct?
• If reported in M4 but not M5 Last time I
  recorded that you received food stamps in
  month 4. Did you continue to receive any
  benefits from food stamps after month 5 1st?
• Health Insurance
• Last time I recorded NAME was covered by
  Medicaid. Is NAME still covered by that type
  of insurance plan?

20
Assessing Dependent Interview Seam-Bias Reduction
Techniques

• Month-to-month transitions in recipiency/enrollmen
  t/coverage (yes/no)
• Transitions at the seams (between M4-M5) where
  fewer transitions improvement
• Calculated of month-to-month transitions for
  unearned income, school enrollment, and health
  insurance coverage

21
Expected of transitions if no seam bias
22
Expected of transitions if no seam bias
23
(No Transcript)
24
Expected of transitions if no seam bias
25
(No Transcript)
26
Conclusions

• D.I. In 2004 SIPP represents major departure from
  past practice
• Thorough testing program suggest
• Substantially improved interview experience from
  Interviewer perspective
• Respondents had few concerns with D.I.
  (qualitative tests)
• R.I.P didnt have big impact on D.I. (3-5)

27
Conclusions

• D.I. beginning to impact attrition?
• D.I. Nonresponse follow-up techniques successful
• Decreased item nonresponse for earning, asset,
  and unearned income amounts
• D.I. techniques helped to reduce seam bias in
  income recipiency enrollment
• D.I. helped reduce on/off seam transitions?
• Still much room for improvement
• SIPP D.I. techniques first fielded in June 2004