The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates - PowerPoint PPT Presentation

Loading...

PPT – The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates PowerPoint presentation | free to download - id: 1f9373-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

Description:

Changes in format affecting the 2002 Economic Census. Paper size: Legal Letter ... Business Survey Participation. Out of Survey Organization Control ... – PowerPoint PPT presentation

Number of Views:23
Avg rating:3.0/5.0
Slides: 52
Provided by: econom46
Learn more at: http://www.amstat.org
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates


1
(No Transcript)
2
The Effects of Survey Design Features and
Economic Conditions on Business Survey Response
Rates
  • Diane K. Willimack
  • U.S. Census Bureau

3
Outline
  • Motivation
  • Economic Census
  • Conceptual framework for nonresponse in
    establishment surveys
  • Data analysis, results, and conclusions
  • Number of pages
  • Economic conditions

4
U.S. Economic Census
  • Establishment level data
  • Collected every 5 years
  • Reference year ends in 2 or 7
  • Collection year is the following year

5
U.S. Economic Census continued
  • Self-administered
  • Paper mail-out/mail-back
  • 2002 Electronic via downloadable software
  • Tailored by industry
  • 2002 520 different questionnaire versions

6
Motivation
  • Changes in format affecting the 2002 Economic
    Census
  • Paper size Legal ? Letter
  • Page layout 2 columns ? 1 column
  • Number of pages increased
  • Response rates decline???

7
Research Question
  1. Was the effect of the number of pages on response
    rates different for the 2002 Economic Census than
    for previous censuses?

8
The Data
  • 1992, 1997, and 2002 Economic Censuses
  • Check-in rates
  • Receipts 2nd time Undeliverables
    Mail Counts
  • 1st week of June, 1993, 1998, 2003
  • Unit of analysis questionnaire version
  • n 1459 questionnaire versions

9
Check-In Rates by Year, 1st week of June of the
Collection Year
Census Year Number of Pages (Range) Mean Check-in Rates Across All Versions
1992 2 14 77.34
1997 2 16 69.50
2002 3 27 65.19
10
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
11
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
12
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
13
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
14
Research Question
  1. Was the effect of the number of pages on response
    rates different for the 2002 Economic Census than
    for previous censuses?

15
A Conceptual Framework for Business Survey
Participation
Out of Survey Organization Control
Under Survey Organization Control
External Environment
Survey Design
The Business
The Respondent
Business Goals
Response Burden
Survey Participation Decision
Source Willimack, D.K., Nichols, E., and Sudman,
S., (2002), Understanding Unit and Item
Nonresponse in Business Surveys, in Survey
Nonresponse, Groves, et al. (eds.) New York
Wiley.
16
Defining the Model
  • Concept
  • (External Environment
  • Characteristics of the Business
  • Characteristics of the Respondent
  • Survey Design)
  • Measure
  • Economic Conditions ?GDP, ?Employment
  • Industrial Sector
  • ----------------------
  • pp for the form, availability of electronic mode

Response Rate f
17
The Model
  • Concept
  • (External Environment
  • Characteristics of the Business
  • Characteristics of the Respondent
  • Survey Design)
  • Measure
  • Economic Conditions ?GDP, ?Employment
  • Industrial Sector
  • ----------------------
  • pp for the form, availability of electronic mode

Response Rate f
18
Main Effects Model What is the relationship,
regardless of the census year?

Response Rate a b (pages)
effects of economic variables gi
(Sector) h (Mode) error term
The omitted variable among the Sector dummies
is Mining.
19
Care in Interpreting Results
  • Behavioral model
  • Explanatory, NOT predictive
  • Estimated coefficients
  • Relative magnitude
  • Direction ( or -)
  • Significance

20
Main Effects Model
Estimated Coefficient Significantly different from 0? p-value
Intercept 81.81 Yes lt.0001
Pages -1.13 Yes lt.0001
Availability of Electronic Mode -6.91 Yes lt.0001

21
Research Question
  • Does 2002 differ from previous censuses?
  • H0 1992 1997 2002
  • Pairwise comparisons

22
Interaction Effects Model Does the effect of
pages differ by census year?

a b92(pp) c92(pp)2 d92 (pp)3
b97(pp) c97(pp)2 d97 (pp)3
b02(pp) c02(pp)2 d02 (pp)3
effect of economic variables gi
(Sector) h (Mode) error
term
Response Rate
The omitted variable among the Sector dummies
is Mining.
23
Does the effect of pages differ by census years?
Testing H0 1992 1997 2002
Reject H0 F-stat p-value
pp H0 b92 b97 b02 Yes 27.23 plt.0001
(pp)2 H0 c92 c97 c02 Yes 18.28 plt.0001
(pp)3 H0 d92 d97 d02 Yes 12.04 plt.0001
24
Interaction Effects Model Does the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94
1997 -11.92
2002 -5.78
(pp)2 1992 1.14
1997 1.23
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
25
Interaction Effects Model Does the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14
1997 1.23
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
26
Interaction Effects Model Does the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14 p.5278 plt.0001
1997 1.23 plt.0001
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
27
Interaction Effects Model Does the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14 p.5278 plt.0001
1997 1.23 plt.0001
2002 0.33
(pp)3 1992 -0.04 p.7564 plt.0001
1997 -0.04 plt.0001
2002 -0.01
28
Estimated Model 1992
Response Rate
Number of Pages
29
Estimated Model 1997
Response Rate
Number of Pages
30
Estimated Model 2002
Response Rate
Number of Pages
31
Rates of Change in Response Rates per Added Page
  • 1992
  • Rate of Change -9.94 2.28(pp) 0.12(pp)2
  • 1997
  • Rate of Change -11.92 2.46(pp) 0.12(pp)2
  • 2002
  • Rate of Change -5.78 0. 66(pp) 0.03(pp)2

32
Research Questions
  1. Was the effect of the number of pages on response
    rates different for the 2002 Economic Census than
    for previous censuses?

33
Research Question 1 Conclusion
  • Negative effect
  • pages ? ? response rates ?
  • Caution Behavioral model vs. Controlled
    Experiment
  • Cannot identify causality
  • Cannot isolate factors

34
Research Question 1 Conclusion continued
  • Reject H0 1992 1997 2002
  • 2002 differed from the previous censuses (per
    results from pairwise comparisons)
  • Negative effect of pages on economic
    census response appears to have
    diminished over time

35
A Conceptual Framework for Business Survey
Participation
Out of Survey Organization Control
Under Survey Organization Control
External Environment
Survey Design
The Business
The Respondent
Business Goals
Response Burden
Survey Participation Decision
Source Willimack, D.K., Nichols, E., and Sudman,
S., (2002), Understanding Unit and Item
Nonresponse in Business Surveys, in Survey
Nonresponse, Groves, et al. (eds.) New York
Wiley.
36
Household Survey Literature
  • Better economic times ? refusal rates increased
    (in a household survey)
  • Harris-Kojetin and Tucker, 1999, Exploring the
    Relation of Economic and Political Conditions
    with Refusal Rates to a Government Survey, JOS,
    15167-184

37
Research Questions continued
  1. Was the effect of the number of pages on response
    rates different for the 2002 Economic Census than
    for previous censuses?
  2. Did economic conditions affect economic census
    response rates?

38
The Model
  • Concept
  • (External Environment
  • Characteristics of the Business
  • Characteristics of the Respondent
  • Survey Design)
  • Measure
  • Economic Conditions ?GDP, ?Employment
  • Industrial Sector
  • ----------------------
  • pp for the form,
  • availability of electronic mode

Response Rate f
39
Main Effects Model What is the relationship,
regardless of the census year?

Response Rate a b (pages) e
(?GDP) f (?Employment) gi
(Sector) h (Mode) error term
The omitted variable among the Sector dummies
is Mining.
40
Main Effects Model
Estimated Coefficient Significantly different from 0? p-value

?GDP -0.12 No 0.4193
?Employment -1.11 Yes lt.0001

41
Interaction Effects Model Do the effects of
economic conditions differ by census year?

a effect of page variables
e92 (?GDP) f92 (?Employment) e97
(?GDP) f97 (?Employment) e02
(?GDP) f02 (?Employment) gi
(Sector) h (Mode) error
term
Response Rate
The omitted variable among the Sector dummies
is Mining.
42
Do the effects of economic conditions differ by
census year? Testing H0 1992 1997 2002
H0 1992 1997 2002 Reject H0 F-stat (p-value)
?GDP No 0.65 (0.5209)
?Employment No 1.25 (0.2861)
43
Research Questions
  1. Was the effect of the number of pages on response
    rates different for the 2002 Economic Census than
    for previous censuses?
  2. Did economic conditions affect economic census
    response rates?

44
Research Question
  • Does the effect of number of pages on response
    rates vary with economic conditions?

45
Mixed Effects Models
For each census year (i 1992, 1997, 2002)
a bi (pp) ci (pp)2 di(pp)3 ei
(? GDP) fi (? Emply) g (Sector) h (Mode)

Response Rate
ji (pp) (? GDP) ki (pp)2 (? GDP) mi
(pp) (? Emply) ni (pp)2 (? Emply) ?
The omitted variable among the Sector dummies
is Mining.
46
Summary of Results Under the Mixed Effects Model
  • Variables that become non-significant (with
    p-values)
  • (pp)2 for 1992 (0.1159)
  • (pp)3 for 2002 (0.3753)

47
Summary of Results Under the Mixed Effects
Model continued
Economic variables that are significant (all are
for 2002) (with p-values)
  • ?GDP (0.0282)
  • (pp) (?GDP) (0.0103)
  • (pp)2 (?GDP) (0.0087)
  • ?Emply (0.0322)
  • (pp) (?Emply ) (0.0117)
  • (pp)2 (?Emply ) (0.0082)

48
What was happening in the U.S. economy in
2001-2003?
  • Economic recovery beginning to emerge from a
    recession.

49
(No Transcript)
50
What was happening in the U.S. economy in
2001-2003?
  • Economic recovery beginning to emerge from a
    recession.
  • ? GDP increasing sharply
  • ? Employment declining, bottoming out
  • Unstable economy
  • Sharp opposite movements in GDP and Employment

51
What is the nature of the interaction between
economic conditions and pages?
  • ?GDP and ?Employment effects tend to be
    offsetting.
  • ?GDP moderates the page effect on response rates.
  • ?Emply exacerbates the page effect on response
    rates.
  • Magnitude of Employment effect is larger.

52
What is the nature of the interaction between
economic conditions and pages? continued
  • Magnitude of Employment effect is larger.
  • The more negative the change in employment, the
    more negative is the effect of pages on response
    rates.
  • The curve representing the relationship between
    pages and response rates would be flatter.

53
What is the nature of the interaction between
economic conditions and pages? continued
Employment effect
Response Rates
Number of pages
54
Economic Effects Postscript Economic Theory of
Lagged Expectations
  • In-depth examination of effects of quarterly
    employment change on response rates
  • Statistically significant impact of changes 4
    quarters and 3 quarters prior to beginning of
    economic census data collection.

55
Interpreting the Economic Effect
  • Unstable economy
  • Employment decline and GDP growth
  • Uncertainty
  • Riskier to divert resources to non-productive
    activities
  • Cost-benefit
  • Survey response results in a cost with no
    associated productive benefit.

56
Interpreting the Economic Effect continued
  • The bottom line
  • In a risky economic environment
  • Employees are too valuable to devote to filling
    out forms.
  • Forms that appear longer (pages) are presumed to
    take longer and be more costly to complete
  • Response rates are in jeopardy!

57
Summary Number of Pages
  • Negative effect
  • pages ? ? response rates ?
  • Cubic functional relationship
  • Negative effect of pages on economic
    census response appears to have
    diminished over time

58
Summary Economic Conditions
  • Evidence that economic conditions interact with
    form length to affect response rates
  • Mainly through changes in employment
  • Survey response non-productive cost
  • Corroborates qualitative research findings
About PowerShow.com