Representativity Indicators for Survey Quality

1

• Representativity Indicators for Survey Quality
• Programme Cooperation
• Theme Socio-economic sciences and Humanities
• Activity Socio-economic and scientific
  indicators
• Area Provision for underlying official
  statistics
• Costs 827,000
• Length 28 months (from 1-3-2008 to 30-06-2010)

2
Is the response rate a good quality indicator?
3
Is more response always better?

• The accuracy of survey estimates is determined by
  the precision (variance) and the bias of
  estimators.
• A higher response rate is only better if the bias
  is smaller. This is not always the case!
• Integrated Survey on Household Living Conditions
  1998
• The composition of the sample deteriorated in
  month 2.

4
Is more response always better?

• Other examples
• The composition of the response deteriorates
  after a call-back survey among non-respondents.
• The composition of the response may deteriorate
  if incentives are used to increase response.

5
There is a need for better quality indicators

• Such quality indicators should reflect how well
  the composition of the survey response reflects
  the population (or complete sample).
• The RISQ project intends to develop and test such
  indicators.
• These indicators will be called R-indicators
  (short for Representativity Indicators).
• Such indicators should be comparable over space
  (countries, regions) and time.
• They should be useful in the data collection
  phase and the analysis phase of the survey.
• They should also be useful for processing
  register data.
• They should not depend on the survey variables.

6
An example of an R-indicator

• The bias (due to non-response) of the response
  mean is equal to
• Y is the survey variable and ? is the response
  probability.
• Cor(Y, ?) is the correlation between Y and ?.
• S(Y) and S(?) are the standard deviations of Y
  and ?.
• The bias vanishes if all response probabilities
  are equal. Then

7
An example of an R-indicator (continued)

• Definition of an indicator
• M(?) 1 All response probabilities are equal.
  The response is representative.
• M(?) 0 Maximum possible deviation from
  representativity.
• Computation of M(?)
• Required auxiliary variables.
• Fit logit (or other) model for response
  probabilities.
• Estimate response probabilities.

8
Research issues

• Development of R-indicators based on variance of
  response probabilities.
• Development of R-indicators based on g-weights.
• Analysis of dependency on sample size.
• Analysis of dependency on auxiliary variables.
• Estimation of response probabilities if only
  population distribution of auxiliary variables is
  available.
• Development of partial R-indicators to identify
  groups at risk.
• Use of paradata (fieldwork data) in response
  probability models.

9
An example of an R-indicator (continued)

• Example Dutch Labour Force Survey.
• Sample of non-respondents re-approached
  (call-back) with complete questionnaire.
• Sample of non-respondents re-approached with
  small questionnaire (basic question approach).
• The composition of the response improves more
  after the call-back approach.

10
The RISQ Project

• Objectives of the project
• Develop R-indicators based on variance of
  response probabilities and g-weights.
• Determine statistical properties of indicators.
• Develop tools to compute indicators.
• Explore use for monitoring data collection.
• Explore use for controlling data collection.
• Explore use in correction (selection of weighting
  variables).
• Explore use in analysis (space and time).
• Test on real data sets (surveys and registers).
• Social and economic surveys.

11
The RISQ Project

• Project partners
• Statistics Netherlands (Netherlands,
  co-ordinator)
• University of Southampton (UK)
• Statistics Norway (Norway)
• University of Leuven (Belgium)
• Statistical Office of the Republic of Slovenia
  (Slovenia)
• Website
• www.r-indicator.eu