Nonresponse Bias in the 2003 Survey of Small Business Finances

1
Nonresponse Bias in the 2003 Survey of Small
Business Finances

• Y. Michael Yang, NORC at the University of
  Chicago
• Traci Mach, Federal Reserve Board
• Lieu Hazelwood, Federal Reserve Board

2
Disclaimer

• The views expressed herein are those of the
  authors. They do not necessarily reflect the
  opinions of the Federal Reserve Board or its
  staff.

3
Overview

• Bias due to Nonresponse
• Data Source 2003 Survey of Small Business
  Finances
• Analytical Methods for Assessing Bias
• Results
• Discussions

4
Bias Due to Nonresponse (1)

• Deterministic model
• Probabilistic model

5
Bias Due to Nonresponse (2)

• Implications
• Nonresponse bias is variable-specific
• Nonresponse does not necessarily lead to bias
• Nonresponse bias increases as response rate
  decreases
• Nonresponse bias depends on the response
  mechanism
• Key is to examine the correlation between
  response propensity and the survey variable

6
2003 SSBF (1)

• Target population
• U.S. for-profit small businesses
• Frame
• Duns Market Identifier (DMI)
• Sample design
• Stratified systematic sampling
• Screener nonresponse subsampling
• Main interview nonresponse subsampling
• Data collection
• Telephone Interview

7
2003 SSBF (2)
8
2003 SSBF (3)

• Scope of analysis
• Eligible screener respondents (n6,520)
• Focus on bias due to main interview nonresponse
• Adjustments for screener nonresponse and
  subsampling have been incorporated in the base
  weight

9
Analytical Methods

• Comparing response rates across subgroups
• Comparing early responders and late responders on
  survey variables
• Comparing first phase responders and follow-up
  responders on survey variables
• Comparing unadjusted and adjusted estimates
• Estimate nonresponse bias analytically

10
Main Interview Outcome Rates
11
Outcome Rates by Business Size Unweighted
12
Outcome Rates by Business Size Weighted
13
Outcome Rates by Urbanicity Unweighted
14
Outcome Rates by Urbanicity Weighted
15
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Unweighted (1)
16
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Unweighted (2)
17
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Unweighted (3)
18
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Weighted (1)
19
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Weighted (2)
20
Early (lt10 calls) Vs. Late (gt10 calls)
Completes Weighted (3)
21
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Unweighted (1)
22
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Unweighted (2)
23
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Unweighted (3)
24
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Weighted (1)
25
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Weighted (2)
26
Early (lt19 calls) Vs. Late (gt19 calls)
Completes Weighted (3)
27
Main Sample Vs. Follow-up Subsample Completes
Unweighted (1)
28
Main Sample Vs. Follow-up Subsample Completes
Unweighted (2)
29
Main Sample Vs. Follow-up Subsample Completes
Unweighted (3)
30
Main Sample Vs. Follow-up Subsample Completes
Weighted (1)
31
Main Sample Vs. Follow-up Subsample Completes
Weighted (2)
32
Main Sample Vs. Follow-up Subsample Completes
Weighted (3)
33
Indirect Estimate of Bias

• (unadjusted estimate)-(adjusted estimate)
• Measures the amount of nonresponse adjustment
• Measures nonresponse bias if the adjusted
  estimate is unbiased

34
Indirect Estimates of Bias
Characteristics Estimated Bias Characteristics Estimated Bias
Native American 0.14 Equipment loans -1.27
White 0.07 Capital leases -0.18
White non-Hispanic 0.24 Other loans -0.38
Asian -0.13 Line of credit -0.65
Asian or Pacific Islander -0.1 Transaction service -0.57
Black 0.04 Cash management service -0.51
Pacific Islander 0.03 Credit related service -0.53
Hispanic -0.12 Trust service -0.71
Owner managed 0.77 Brokerage service -0.16
Average owners education 0.02 Card processing service -0.33
Firm age 0.07 Total sales -158570.05
Minority or Hispanic -0.1 Profit -20781.18
Total employment -1.01 Total Assets -77824.28
Mortgage loans 0.07 DB credit percentile -0.96
Motor vehicle loans -1.02 Total employees -1.31
35
Direct Estimate of Bias

• Estimate response propensity
• Compute the covariance between response
  propensity and survey variable
• Estimate bias analytically

36
Direct Estimates of Bias (1)
Survey Variable Covariance Estimated Bias Weighted Estimate Bias Ratio
Native American 0.01 0.02 1.41 1.4
White 0.08 0.16 90.82 0.2
White non-Hispanic 0.15 0.31 86.43 0.4
Asian -0.05 -0.11 4.17 -2.6
Asian or Pacific Islander -0.06 -0.11 4.29 -2.6
Black -0.04 -0.07 3.63 -1.9
Pacific Islander 0 0 0.12 0.0
Hispanic -0.09 -0.18 4.28 -4.2
Owner managed 0.15 0.29 94.16 0.3
Average owners education 0.01 0.01 4.61 0.2
Firm age -0.03 -0.05 14.3 -0.3
Minority or Hispanic -0.17 -0.35 13.31 -2.6
Total employment -1.91 -3.85 8.73 -44.1
Mortgage loans -0.11 -0.22 13 -1.7
Motor vehicle loans -0.28 -0.57 25.33 -2.3
37
Direct Estimates of Bias Bias (2)
Survey Variable Covariance Estimated Bias Weighted Estimate Bias Ratio
Equipment loans -0.27 -0.54 10.59 -5.1
Capital leases -0.21 -0.42 8.9 -4.7
Other loans -0.21 -0.41 10.08 -4.1
Line of credit -0.53 -1.08 32.45 -3.3
Transaction service -0.4 -0.8 38.96 -2.1
Cash management service -0.55 -1.11 6.43 -17.3
Credit related service -0.25 -0.5 4.82 -10.4
Trust service -0.45 -0.92 17.62 -5.2
Brokerage service -0.05 -0.11 5.45 -2.0
Card processing service -0.16 -0.31 36.67 -0.8
Total sales -331788.2 -669119.94 1,177,572 -56.8
Profit -31853.62 -64239.45 175,965 -36.5
Total Assets -156104.5 -314817.2 535,133 -58.8
38
Direct vs. Indirect Bias Estimates
39
Discussion

• There is significant correlation between
  nonresponse and key survey variables
• Larger and urban businesses have lower response
  rate
• Larger and urban businesses use more services,
  have more sales, assets, and profit
• Unadjusted estimates of the mean/total are biased
  downward for most variables
• Response propensity-based weighting appears to be
  effective, but only a portion of the bias is
  corrected
• Weighting method may be improved in future
  surveys

40
Contact Information

• Yang-michael_at_norc.org