The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

1
(No Transcript)
2
The Effects of Survey Design Features and
Economic Conditions on Business Survey Response
Rates

• Diane K. Willimack
• U.S. Census Bureau

3
Outline

• Motivation
• Economic Census
• Conceptual framework for nonresponse in
  establishment surveys
• Data analysis, results, and conclusions
• Number of pages
• Economic conditions

4
U.S. Economic Census

• Establishment level data
• Collected every 5 years
• Reference year ends in 2 or 7
• Collection year is the following year

5
U.S. Economic Census continued

• Self-administered
• Paper mail-out/mail-back
• 2002 Electronic via downloadable software
• Tailored by industry
• 2002 520 different questionnaire versions

6
Motivation

• Changes in format affecting the 2002 Economic
  Census
• Paper size Legal ? Letter
• Page layout 2 columns ? 1 column
• Number of pages increased
• Response rates decline???

7
Research Question

1. Was the effect of the number of pages on response
   rates different for the 2002 Economic Census than
   for previous censuses?

8
The Data

• 1992, 1997, and 2002 Economic Censuses
• Check-in rates
• Receipts 2nd time Undeliverables
  Mail Counts
• 1st week of June, 1993, 1998, 2003
• Unit of analysis questionnaire version
• n 1459 questionnaire versions

9
Check-In Rates by Year,1st week of June of the
Collection Year
Census Year Number of Pages (Range) Mean Check-in Rates Across All Versions
1992 2 14 77.34
1997 2 16 69.50
2002 3 27 65.19
10
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
11
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
12
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
13
Industry-Level Summary
Check-in Rate Pages Correlation
Industry Mean Mean (resprate, pp)
Mining 67.91 10.26 -0.4683
Manufacturing 68.71 9.79 -0.4661
Retail 70.09 5.14 -0.5225
Utilities 71.71 4.48 -0.5168
Wholesale 73.46 6.35 -0.6577
Construction 73.88 6.07 -0.5784
Fin, Ins, Real Est 75.01 5.33 -0.2967
Services 75.44 4.95 -0.6239
Overall 70.59 8.90 -0.6061
All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05. All correlations are statistically significant at ?.05.
14
Research Question

1. Was the effect of the number of pages on response
   rates different for the 2002 Economic Census than
   for previous censuses?

15
A Conceptual Framework forBusiness Survey
Participation
Out of Survey Organization Control
Under Survey Organization Control
External Environment
Survey Design
The Business
The Respondent
Business Goals
Response Burden
Survey Participation Decision
Source Willimack, D.K., Nichols, E., and Sudman,
S., (2002), Understanding Unit and Item
Nonresponse in Business Surveys, in Survey
Nonresponse, Groves, et al. (eds.) New York
Wiley.
16
Defining the Model

• Concept
• (External Environment
• Characteristics of the Business
• Characteristics of the Respondent
• Survey Design)

• Measure
• Economic Conditions ?GDP, ?Employment
• Industrial Sector
• ----------------------
• pp for the form, availability of electronic mode

Response Rate f
17
The Model

• Concept
• (External Environment
• Characteristics of the Business
• Characteristics of the Respondent
• Survey Design)

• Measure
• Economic Conditions ?GDP, ?Employment
• Industrial Sector
• ----------------------
• pp for the form, availability of electronic mode

Response Rate f
18
Main Effects ModelWhat is the relationship,
regardless of the census year?

Response Rate a b (pages)
effects of economic variables gi
(Sector) h (Mode) error term
The omitted variable among the Sector dummies
is Mining.
19
Care in Interpreting Results

• Behavioral model
• Explanatory, NOT predictive
• Estimated coefficients
• Relative magnitude
• Direction ( or -)
• Significance

20
Main Effects Model
Estimated Coefficient Significantly different from 0? p-value
Intercept 81.81 Yes lt.0001
Pages -1.13 Yes lt.0001
Availability of Electronic Mode -6.91 Yes lt.0001

21
Research Question

• Does 2002 differ from previous censuses?
• H0 1992 1997 2002
• Pairwise comparisons

22
Interaction Effects ModelDoes the effect of
pages differ by census year?

a b92(pp) c92(pp)2 d92 (pp)3
b97(pp) c97(pp)2 d97 (pp)3
b02(pp) c02(pp)2 d02 (pp)3
effect of economic variables gi
(Sector) h (Mode) error
term
Response Rate
The omitted variable among the Sector dummies
is Mining.
23
Does the effect of pages differ by census years?
Testing H0 1992 1997 2002
Reject H0 F-stat p-value
pp H0 b92 b97 b02 Yes 27.23 plt.0001
(pp)2 H0 c92 c97 c02 Yes 18.28 plt.0001
(pp)3 H0 d92 d97 d02 Yes 12.04 plt.0001
24
Interaction Effects ModelDoes the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94
1997 -11.92
2002 -5.78
(pp)2 1992 1.14
1997 1.23
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
25
Interaction Effects ModelDoes the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14
1997 1.23
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
26
Interaction Effects ModelDoes the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14 p.5278 plt.0001
1997 1.23 plt.0001
2002 0.33
(pp)3 1992 -0.04
1997 -0.04
2002 -0.01
27
Interaction Effects ModelDoes the effect of
pages differ by census year?
Page Term Year Estimated Coefficient Differ from 1997? Differ from 2002?
pp 1992 -9.94 p.0028 plt.0001
1997 -11.92 plt.0001
2002 -5.78
(pp)2 1992 1.14 p.5278 plt.0001
1997 1.23 plt.0001
2002 0.33
(pp)3 1992 -0.04 p.7564 plt.0001
1997 -0.04 plt.0001
2002 -0.01
28
Estimated Model 1992
Response Rate
Number of Pages
29
Estimated Model 1997
Response Rate
Number of Pages
30
Estimated Model 2002
Response Rate
Number of Pages
31
Rates of Change in Response Rates per Added Page

• 1992
• Rate of Change -9.94 2.28(pp) 0.12(pp)2
• 1997
• Rate of Change -11.92 2.46(pp) 0.12(pp)2
• 2002
• Rate of Change -5.78 0. 66(pp) 0.03(pp)2

32
Research Questions

1. Was the effect of the number of pages on response
   rates different for the 2002 Economic Census than
   for previous censuses?

33
Research Question 1 Conclusion

• Negative effect
• pages ? ? response rates ?
• Caution Behavioral model vs. Controlled
  Experiment
• Cannot identify causality
• Cannot isolate factors

34
Research Question 1 Conclusioncontinued

• Reject H0 1992 1997 2002
• 2002 differed from the previous censuses (per
  results from pairwise comparisons)
• Negative effect of pages on economic
  census response appears to have
  diminished over time

35
A Conceptual Framework forBusiness Survey
Participation
Out of Survey Organization Control
Under Survey Organization Control
External Environment
Survey Design
The Business
The Respondent
Business Goals
Response Burden
Survey Participation Decision
Source Willimack, D.K., Nichols, E., and Sudman,
S., (2002), Understanding Unit and Item
Nonresponse in Business Surveys, in Survey
Nonresponse, Groves, et al. (eds.) New York
Wiley.
36
Household SurveyLiterature

• Better economic times ? refusal rates increased
  (in a household survey)
• Harris-Kojetin and Tucker, 1999, Exploring the
  Relation of Economic and Political Conditions
  with Refusal Rates to a Government Survey, JOS,
  15167-184

37
Research Questionscontinued

1. Was the effect of the number of pages on response
   rates different for the 2002 Economic Census than
   for previous censuses?
2. Did economic conditions affect economic census
   response rates?

38
The Model

• Concept
• (External Environment
• Characteristics of the Business
• Characteristics of the Respondent
• Survey Design)

• Measure
• Economic Conditions ?GDP, ?Employment
• Industrial Sector
• ----------------------
• pp for the form,
• availability of electronic mode

Response Rate f
39
Main Effects ModelWhat is the relationship,
regardless of the census year?

Response Rate a b (pages) e
(?GDP) f (?Employment) gi
(Sector) h (Mode) error term
The omitted variable among the Sector dummies
is Mining.
40
Main Effects Model
Estimated Coefficient Significantly different from 0? p-value

?GDP -0.12 No 0.4193
?Employment -1.11 Yes lt.0001

41
Interaction Effects ModelDo the effects of
economic conditions differ by census year?

a effect of page variables
e92 (?GDP) f92 (?Employment) e97
(?GDP) f97 (?Employment) e02
(?GDP) f02 (?Employment) gi
(Sector) h (Mode) error
term
Response Rate
The omitted variable among the Sector dummies
is Mining.
42
Do the effects of economic conditions differ by
census year? Testing H0 1992 1997 2002
H0 1992 1997 2002 Reject H0 F-stat (p-value)
?GDP No 0.65 (0.5209)
?Employment No 1.25 (0.2861)
43
Research Questions

1. Was the effect of the number of pages on response
   rates different for the 2002 Economic Census than
   for previous censuses?
2. Did economic conditions affect economic census
   response rates?

44
Research Question

• Does the effect of number of pages on response
  rates vary with economic conditions?

45
Mixed Effects Models
For each census year (i 1992, 1997, 2002)
a bi (pp) ci (pp)2 di(pp)3 ei
(? GDP) fi (? Emply) g (Sector) h (Mode)

Response Rate
ji (pp) (? GDP) ki (pp)2 (? GDP) mi
(pp) (? Emply) ni (pp)2 (? Emply) ?
The omitted variable among the Sector dummies
is Mining.
46
Summary of Results Under the Mixed Effects Model

• Variables that become non-significant (with
  p-values)
• (pp)2 for 1992 (0.1159)
• (pp)3 for 2002 (0.3753)

47
Summary of Results Under the Mixed Effects
Modelcontinued
Economic variables that are significant (all are
for 2002) (with p-values)

• ?GDP (0.0282)
• (pp) (?GDP) (0.0103)
• (pp)2 (?GDP) (0.0087)

• ?Emply (0.0322)
• (pp) (?Emply ) (0.0117)
• (pp)2 (?Emply ) (0.0082)

48
What was happening in the U.S. economy in
2001-2003?

• Economic recovery beginning to emerge from a
  recession.

49
(No Transcript)
50
What was happening in the U.S. economy in
2001-2003?

• Economic recovery beginning to emerge from a
  recession.
• ? GDP increasing sharply
• ? Employment declining, bottoming out
• Unstable economy
• Sharp opposite movements in GDP and Employment

51
What is the nature of the interaction between
economic conditions and pages?

• ?GDP and ?Employment effects tend to be
  offsetting.
• ?GDP moderates the page effect on response rates.
• ?Emply exacerbates the page effect on response
  rates.
• Magnitude of Employment effect is larger.

52
What is the nature of the interaction between
economic conditions and pages? continued

• Magnitude of Employment effect is larger.
• The more negative the change in employment, the
  more negative is the effect of pages on response
  rates.
• The curve representing the relationship between
  pages and response rates would be flatter.

53
What is the nature of the interaction between
economic conditions and pages? continued
Employment effect
Response Rates
Number of pages
54
Economic Effects Postscript Economic Theory of
Lagged Expectations

• In-depth examination of effects of quarterly
  employment change on response rates
• Statistically significant impact of changes 4
  quarters and 3 quarters prior to beginning of
  economic census data collection.

55
Interpreting the Economic Effect

• Unstable economy
• Employment decline and GDP growth
• Uncertainty
• Riskier to divert resources to non-productive
  activities
• Cost-benefit
• Survey response results in a cost with no
  associated productive benefit.

56
Interpreting the Economic Effect continued

• The bottom line
• In a risky economic environment
• Employees are too valuable to devote to filling
  out forms.
• Forms that appear longer (pages) are presumed to
  take longer and be more costly to complete
• Response rates are in jeopardy!

57
SummaryNumber of Pages

• Negative effect
• pages ? ? response rates ?
• Cubic functional relationship
• Negative effect of pages on economic
  census response appears to have
  diminished over time

58
SummaryEconomic Conditions

• Evidence that economic conditions interact with
  form length to affect response rates
• Mainly through changes in employment
• Survey response non-productive cost
• Corroborates qualitative research findings