Responsive%20Design%20for%20Household%20Surveys:%20Illustration%20of%20Management%20Interventions%20Based%20on%20Survey%20Paradata - PowerPoint PPT Presentation

About This Presentation
Title:

Responsive%20Design%20for%20Household%20Surveys:%20Illustration%20of%20Management%20Interventions%20Based%20on%20Survey%20Paradata

Description:

Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis, James ... – PowerPoint PPT presentation

Number of Views:138
Avg rating:3.0/5.0

less

Transcript and Presenter's Notes

Title: Responsive%20Design%20for%20Household%20Surveys:%20Illustration%20of%20Management%20Interventions%20Based%20on%20Survey%20Paradata


1
Responsive Design for Household Surveys
Illustration of Management Interventions Based
on Survey Paradata
  • Robert M. Groves, Emilia Peytcheva, Nicole
    Kirgis, James Wagner, William Axinn, University
    of Michigan, USAWilliam Mosher, US National
    Center for Health Statistics

Research partially supported by contract with the
US National Center for Health Statistics,
Contract No. 200-2000-07001
2
Definition Responsive Design
Survey designs that
  1. Preidentify a set of alternative features
    potentially affecting costs and errors of
    statistics
  2. Identify a set of indicators of the cost and
    error properties of those feature
  3. Monitor indicators in initial stages of data
    collection
  4. Alter the active features of the survey based on
    cost/error tradeoff decision rules
  5. Combine data from separate phases into a single
    estimator

3
The Whys of Responsive Designs
  • One-off surveys are mounted with large
    uncertainties (e.g., eligibility of frame
    elements, effort required to contact, cooperation
    rate, length of interview)
  • Most survey budgets are relatively fixed at start
    of project
  • Some survey errors are functions of effort during
    production
  • Hence, quality is out of control of researcher
    unless designs are permitted to change based on
    production experience

4
The NSFG Dashboard
Effort
Active Sample
Productivity
interviews
occupied
Irs working
hours
eligible
cum. interviews
production
nonworked
hours/interview
X
calls/day
calls/interview
noncontacts
mean calls
calls/hour
peak calls
8 calls
Data Set Balance
scrnr/main calls
locked bldgs
resistant
response rate
hard appt.
with kids
propensity
sexually active
group rates
CV group rates
5
Evaluation of Two Interventions Based on Paradata
  • Survey setting face-to-face survey screening
    (3-5 min.) to locate 60 of households with 15-44
    year old one eligible sampled (60-80 min.)
  • Interventions
  • Increasing relative effort on screening
    interviews versus main interviews with selected
    respondent
  • Increasing relative effort on a small subset of
    cases with high selection weights and high
    propensities to respond

6
Ratio of Screener Calls to Main Interview Calls
by Day by Quarter
7
Autoregressive Time Series Coefficients for Model
Predicting Daily Number of Screener
Calls(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 279.9 (lt.0001) 385.0 (lt.0001) 345.3 (lt.0001) 294.8 (lt.0001)
Day -0.73 (.48) -3.2 (0.28) -2.1 (.21) -2.0 (.17)
Screener Week 28.3 (.29) 97.5 (.009) 24.6 (.56) 39.0 (.29)
8
Autoregressive Time Series Coefficients for Model
Predicting Daily Number of Screener
Interviews(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 63.6 (lt.0001) 94.5 (lt.0001) 64.6 (lt.0001) 63.4 (lt.0001)
Day -0.51 (.032) -1.23 (0.001) -0.54 (.16) -0.69 (.09)
Screener Week 6.1 (.30) 18.5 (.01) 12.3 (.20) 6.1 (.54)
9
Second Intervention Increased Emphasis on Subset
of Active Cases
  • In the last weeks of Phase 1, a subsample of
    cases with high propensities and high selection
    weights are identified
  • These cases are chosen to improve balance of
    respondent pool
  • Interviewers are asked to give greater emphasis
    to these cases

10
Mean Expected Probability to be Interviewed on
Next Call, Screeners (Red) and Main (Green) by
Day of Data Collection by Quarter
11
Analytic Approach
  • Not all interviewers active workloads contain
    both intervention cases and non-intervention
    cases
  • We limit the analysis to those who have both
    types of cases
  • We examine two indicators of success
  • Mean number of calls (imperfect)
  • Response rate in intervention period

12
Comparison of Mean Screener Calls During
Intervention Period for Intervention and
Nonintervention Cases
Little evidence of increased calling on
intervention cases
13
Comparison of Screener Response Rate During
Intervention Period for Intervention and
Nonintervention Cases
Little evidence of increased response rate on
intervention cases
14
Comparison of Mean Main Calls During Intervention
Period for Intervention and Nonintervention Cases
Mixed evidence on higher calls on intervention
cases
15
Comparison of Main Response Rate During
Intervention Period for Intervention and
Nonintervention Cases
General tendency to higher response rates for
intervention cases
16
Logistic Regression For Likelihood of Main
Interview
Quarter 2 Quarter 3 Quarter 4 Quarter 5
Intercept -2.13 -1.97 -0.81 -0.84
Intervention -0.07 0.72 0.13 0.50
Weight 0.00 0.00 0.00 0.00
Propensity 5.46 2.98 2.02 0.65
17
Conclusions
  • Intervention 1 Management direction to focus on
    screeners vs. main increases calls, sometimes
    dramatically screener interviews follow
  • Intervention 2 Effectiveness at focusing on
    individual cases greater for main interviews than
    screener interviews

18
Next Steps on Responsive Design with Paradata
  • Responsive design requires effective central
    management direction of interviewer behavior
  • Were still learning how to communicate these
    directives consistently well
Write a Comment
User Comments (0)
About PowerShow.com