Joint ECOECD Workshop on International Development of Business and Consumer Tendency Surveys Brussel

1
Joint EC-OECD Workshop on International
Development of Business and Consumer Tendency
SurveysBrussels 14-15 November 2005Task
Force on Harmonisation of Survey Operation and
Technical Design

• Efficient Sample Design and Weighting
  Methodologies
• Analysis of Key Issues and Recommendations

2
Efficient Sample Design and Weighting
Methodologies

• The usefulness of BTS and CS strongly depends on
  their statistical quality
• Quality of survey data may be measured in terms
  of (OECD, 2003)
• reliability
• timeliness of release
• comparability over time
• transparency and accessibility to the users
• This task force has to do with sample design and
  weighting methodologies
• A sound definition of these aspects increases
  the reliability and therefore the overall
  quality of the data

3
Efficient Sample Design and Weighting
Methodologies

• Business Tendency Surveys (BTS) are conducted on
  manufacturing, services, retail and construction
  sectors
• Consumer Surveys (CS) measure households
  opinion and expectations on personal and general
  economic situation
• For both BTS and CS, key issues in Efficient
  Sampling Design and Weighting Methodologies will
  be analyzed
• An overview of the current practices in these
  fields will be presented, from which it will be
  derived a number of draft recommendations aimed
  at improving the overall quality of the surveys

4
Efficient Sampling Design for Business Tendency
Surveys

• In the case of BTS, key Issues for an efficient
  sample design are (see also Donzè, Etter, Sydow,
  Zellweger, Sample Design for Indystry Survey,
  ECFIN/2003/A3-03)
• Identification of relevant Universe/reference
  population
• Identification of the sample frame
• Identification of the correct method for sample
  selection
• Treatment of missing data

5
Efficient Sampling Design for Business Tendency
Surveys

• The first step in setting up a BTS is the choice
  of the Relevant Universe/reference population
• Typically, it is represented by all the firms
  operating in a given sector, as resulting from
  some official/statistical register
• Some firms may be excluded looking at their size
  (i.e., below a certain size threshold) or
  location, or on the basis of their structural
  characteristic (i.e., exclusion of government
  bodies)

6
Efficient Sampling Design for Business Tendency
Surveys

• The second step is the choice of the sample
  frame, having the goal of maximize sample
  coverage and minimize coverage errors
• BTS are usually based on a panel of responding
  firms, that are re-interviewed each month
• Demographic and structural characteristic of the
  respondents have to be known in order to build up
  the sample, the construction of the frame
  implying the following steps
• identification of the appropriate frame list
• eventual adoption of a cut off strategy
• identification of the sample, reporting and
  response unit
• Updating of the frame list

7
Efficient Sampling Design for Business Tendency
Surveys

• The frame list may be derived from
• official or statistical registers
• membership lists of business associations and
  chamber of commerce

8
Efficient Sampling Design for Business Tendency
Surveys

• The adoption of a cut off strategy may respond
  to the need of
• better focusing on the sector of interest
  (cut-off with respect to the sector of activity)
• ensuring a certain stability of the panel,
  excluding firms below a certain threshold
  (cut-off with respect to size)
• The sample unit is the unit on which to perform
  the sample selection procedure the main choice
  is between having
• the whole firm
• establishments, local units, or kind of activity
  units (kau)
• Even if the firm is chosen as the sample unit,
  it is possible to have more reporting and
  response units within the firm, sending the
  questionnaires to different establishments/local
  units/KAUs within the firm

9
Efficient Sampling Design for Business Tendency
Surveys

10
Efficient Sampling Design for Business Tendency
Surveys

• Finally, the list should be updated frequently
  in order to keep track of the changes in the
  structure of the reference Universe and avoid
  possible problems in terms of
• Under coverage (new firms entering the market)
• ineligibility (old firms exiting the market)
• duplicate entries (errors)

11
Efficient Sampling Design for Business Tendency
Surveys

• The third step implies the choice of an
  appropriate method of sample selection
• The sample should be representative of the
  relevant Universe
• In order to build up a representative sample,
  choices have to be made relatively to
• The sampling method
• The sampling size
• BTS are usually based on a fixed panel of
  reporting units
• A fixed sample structure may rise
  representitiveness problems, because the panel
  may loose its initial representativeness if it is
  not updated regularly
• For this reason, Institutes often use a rotating
  panel method, in which a a fixed percentage of
  units are replaced at regular intervals
• More precisely, the largest and most important
  firms should ideally always be included, with
  smaller firms being rotated out on a regular basis

12
Efficient Sampling Design for Business Tendency
Surveys

• Possible methods of sample selection include
• non-statistically founded methods, such as
• comprehensive surveys with cut-off
• purposive or quota sampling
• statistically funded methods, such as
• simple stratified sampling
• stratified sampling (PPS, OAS)
• Once selecting the sample, Institutes have to
  choose appropriate sample sizes
• Generally speaking, sample size will be chosen
  accordingly to a predetermined desired maximum
  measurement error

13
Efficient Sampling Design for Business Tendency
Surveys

14
Efficient Sampling Design for Business Tendency
Surveys

• The fourth step in building up BTS implies the
  treatment of missing data
• It is possible to distinguish between
• unit non response (entire interview is missing)
• item non response (some answer is missing)
• If missing data occur with large, dominant firms
  the problem is severe
• If there is missing data among the smallest
  firms, the problem is less critical and missing
  data may easily be imputed on the basis of the
  answer of similar firms
• Most commonly used method for dealing with
  missing data problem is the use of follow-up
  techniques (re-interviewing with telephone, fax
  or web based techniques)
• If missing data problems persist, re-weighting
  or imputation methods should be used

15
Weighting Methods for Business Tendency Surveys

• Weighting is used to transform data for the
  realized sample into estimates for the reference
  population
• Weights may be based on
• information coming from the survey itself (size,
  output of the firm)
• auxiliary sources (official/statistical data on
  the size of the reference sector/region)
• Weighting methods are usually based on either
• one stage weight scheme
• two stage weight scheme
• In the one-stage scheme, a weight is associated
  with each reporting unit, in order to take into
  account its relative importance inside the sample
• In the second case, a unit-specific weight is
  used to calculate strata results, further
  aggregating the strata with some external sources
  in order to obtain industry aggregates.

16
Weighting Methods for Business Tendency Surveys

17
Efficient Sampling Design for Consumers Surveys

• In the case of CS, in order to build up a
  representative sample choices have to be made
  relatively to
• The sampling frame
• The sampling methods
• The construction of the sampling frame implies
  the following steps
• identification of the appropriate frame list
• eventual adoption of a cut off strategy
• identification of the sample unit, reporting
  unit and response unit
• updating of the frame list
• In the identification of the appropriate frame
  list, it is crucial that
• the right population is being sampled,
• all the members of the population have the same
  chance of being sampled
• In OECD countries, frame lists are usually based
  on
• official population register (including every
  adult member of the population)
• telephone register (arising possible bias
  problems, to be solved adopting random extraction
  of phone numbers or using other sample techniques
  for those excluded from the directories)

18
Efficient Sampling Design for Consumers Surveys

• A cut off strategy is often adopted
• on the basis of age (cut off age varying often
  across countries in EU)
• in some countries, geographical cut offs are
  also applied
• Response unit may differ from the sample unit
  typically, sample are devised to be
  representative of all households, with the
  selected respondent reporting on the household as
  a whole
• The list should be updated frequently in order
  to monitor as close as possible the evolution of
  the relevant population

19
Efficient Sampling Design for Consumers Surveys

• Key issues in sampling extraction include
• the choice of the appropriate sampling method
• the choice of the optimal size of the sample
• In CS usually a independent cross-section of
  household is extracted each month
• In EU a general strategy of simple random
  sampling is used
• In the US a rotating sampling design is usually
  applied, in which the respondent chosen in each
  drawing is re-interviewed six months later, in
  order to provide a regular assessment of change
  in consumers attitudes
• Most widely used methods of sampling extractions
  include
• simple stratified sampling
• multiple stage stratified sampling
• Random Digit Dialing methods
• There is no consensus in the literature on the
  appropriate sample size
• In practice, sample size currently converges to
  about 2000, a size supposed to provide acceptable
  confidence intervals for this type of survey

20
Weighting Methodologies for Consumers Surveys

• Information gathered from survey respondents
  may be appropriately weighted to derive aggregate
  information on household opinion and
  expectations
• Weights may be based on
• auxiliary information (demographic or
  socio-economic weights)
• inverse selection probabilities (sample weights)
• Most commonly used weight variables are
• demographic characteristics of the household
• gender and age of the respondent
• region of residence and size of the township
• socio economic characteristics of the household
• economic occupation
• level of education
• housing condition, type of area/municipality
• A number of Institutes do not use weights this
  is appropriate only when
• every household has an equal chance of selection
• there is no differential no response

21
Weighting Methodologies for Consumers Surveys

22
Minimum requirements and recommendation for BTS
sample design the sample frame

• The frame lists
• Frame lists should include an as exhaustive as
  possible account of active firms for the survey
  of interest
• As a consequence, the use of official or
  statistical registers of active firms is
  recommended over that of more partial
  business or membership registers
• Cut off strategies
• Institutes are advised to use cut-off strategies
  in order to stabilize the panel (size cut off)
  and for a precise identification of the survey
  objectives (branch cut off)

23
Minimum requirements and recommendation for BTS
sample design the sample frame

• Sample units and reporting units
• Establishments may be considered the ideal
  choice for the sample unit however, it may be
  difficult to gather information at this level
• Use of KAU is advisable if we are particularly
  interested in the industrial structure
• Use of local units is advisable if we are
  particularly interested in the regional structure
  Sample frame reporting units
• Even if the firms is identified as the sample
  unit, it is advisable if possible to have
  different reporting units within the firm
• Response units
• In any case it is strongly recommended that the
  Institutes ensure that the same response unit
  answer the questionnaire every month
• Updating of the lists
• As a minimum requirement, frame lists should be
  updated as soon as a new census of active firms
  is available

24
Minimum requirements and recommendation for BTS
sample design sampling methods

• As a recommendation, a fixed panel should be
  used
• established on a statistically founded basis
• using a rotating pattern of updating
• with a fixed percentage of participants being
  replaced at regular intervals
• As a minimum requirement, sampling extraction
  should be based on sound probabilistic
  considerations
• The use of exhaustive sampling is possible for
  small countries or for a sub-set of the sample
• Avoiding of purposive or ad hoc sampling methods
  is strongly recommended
• Different probabilistic methods of sample
  selection may be used as a general
  consideration, the more heterogeneous is the
  population, the more is advisable the use of
  stratification based sampling methods

25
Minimum requirements and recommendation for BTS
sample design treatment of missing data

• Institutes should define what procedures are
  used for the treatment of item and unit non
  response (missing data)
• As a minimum requirement, institutes are
  advised
• to closely monitor the impact of missing data
  (especially for large firms)
• to use follow up techniques in order to reduce
  their impact (fax, telephone, web remainder)
• The use of imputation methods to deal with
  remaining missing data should be considered with
  care, in order to avoid possible distortions
• The use of re-weighting techniques, taking into
  account different composition of the panel in
  adjacent surveys, may be advisable to reduce the
  bias

26
Minimum requirements and recommendation for BTS
weighting methods

• The use of weights is strongly recommended in
  order to improve the precision of the estimates
• As a minimum requirement the use of a simple
  one stage system of weights is suggested
• Two stage (or multiple stage) weighting
  procedures are advisable for heterogeneous
  population, especially in large countries

27
Minimum requirements and recommendation for CS
sample design the sample frame

• Frame list should include an as exhaustive as
  possible account of the adult population
• As a consequences, official census or
  statistical registers are to be preferred to
  telephone registers
• If telephone registers are used, appropriate
  methods to correct for possible bias is
  recommended
• Cut off strategies with respect to age are
  advisable this may call for further
  harmonization in the EU
• As a recommendation, frame lists should be
  updated yearly

28
Minimum requirements and recommendation for CS
sample design sampling methods

• As a minimum requirement, random sampling
  techniques have to be used in order to ensure
  survey representitiveness
• In case of heterogeneous population, the use of
  stratified sampling methods should be preferred
  to simple random sampling
• Finally, a major difference emerges between EU
  (using independent drawing of the sample each
  month) and the US (using a rotating sample
  design)
• The adoption of the US method may possibly
  enhance research option available to analyst even
  in the EU

29
Minimum requirements and recommendation for CS
weighting

• Weighting is recommended in order to ensure
  better representitiveness
• Demographic characteristic of the households may
  be used as weights, considering among them
• age and gender
• region of residence and size of the township
• Alternatively, socio-economic characteristics
  may be used as weights
• type of occupation
• Level of education
• type of area municipality

30
THANK YOU FOR YOUR ATTENTION!

• Task Force Members are Richard Curtin
  (University of Michigan, United States), Isabelle
  De Greef (National Bank of Belgium), Richard
  Etter (KOF / ETZ, Switzerland), Christian Gayer
  (European Commission), Marie Hormannova (CZSO,
  Czech Republic), Marco Malgarini (Institute for
  Studies and Economic Analysis ISAE, Italy),
  Rony Nilsson (OECD), Raymund Petz (GKI Research,
  Hungary), Takashi Sakuma (ESRI, Japan), Philippe
  Scherrer (INSEE, France), Anna Stangl (Ifo
  Institute for Economic Research, Germany), Andres
  Vertes (GKI Research, Humgary), Peter Weiss
  (European Commission), Jonathan Wood
  (Confederation of British Industry CBI, United
  Kingdom).