Quality strategies in crossnational surveys The case of the European Social Survey

1
Quality strategies in cross-national surveysThe
case of the European Social Survey

• Ineke Stoop

2
Quality and comparability

• Harmonisation strategies and cross-national
  surveys
• Optimal comparability
• European Social Survey
• Nonresponse

3
Harmonisation strategies(Körner and Meyer, 2005)
Harmonised concept
Harmonised concept
Harmonised concept
Measurement procedure
Measurement procedure
Measurement procedure
National survey/ Specific concept
National survey/ Specific concept
National survey/ Specific concept
Input harmonisation
Ex-ante output harmonisation
Ex-post output harmonisation
4
Type of harmonisation

• Ex post output harmonisation
• Use existing sources
• Ex ante output harmonisation
• Set up new survey (or develop new instrument)
• Output harmonisation
• Best national quality
• Or, national survey tradition
• But, what about optimal comparability
• Input harmonisation
• Design new survey
• Optimal comparability
• Or, are identical methods and instruments really
  equivalent in different countries?
• But, what about optimal quality

5
Strategies in comparative survey

• Quality
• Optimal national approach
• Comparability?
• Poor performance accepted
• Some restrictions and standards

• Consistency
• Identical approach
• Lowest standards
• Possible?
• Pull down level
• Highest standards
• Possible?
• Pull up level
• High standards and optimal national approach

6
Mixed approach

• Focus on those key aspects that may hamper
  comparability
• Data collection mode
• Sampling approach
• Response rates
• Question formulation

7
European Social Survey

• Attitudes, values and beliefs
• Bi-annual 2002, 2004, 2006, 2008
• Fixed core, rotating modules
• Effective size 1500 or more
• Central co-ordination
• National implementation
• 30 National Coordinators responsible for
  fieldwork
• Face-to-face

8
Aims European Social Survey

• To monitor and interpret public attitudes and
  values within Europe and to investigate how they
  interact with Europes changing institutions
• Provide data on beliefs, attitudes and values for
  scientific and policy making purposes
• Measure attitude change in a changing Europe

• To advance and consolidate improved methods of
  cross-national survey measurement in Europe and
  beyond
• Collect data according to highest standards
• Generate methodological research
• Develop and disseminate new best practices
• Develop and improve social survey research
  infrastructure in Europe

9
Everything documented

• EVERYTHING AVAILABLE

10
Nonresponse consistency

• Strict probability sampling
• No substitution
• Fieldwork monitoring
• Guidelines response enhancement
• Incentives
• Brochure
• Guidelines interviewer training

• Target noncontact rate 3
• Minimum number of calls
• Timing of calls (evening, weekend)
• Target response rate 70
• Refusal conversion suggested
• Contact form and call records
• Standard response rate calculation

11
Nonresponse divergence

• Population
• Residential population
• Minority languages
• Sampling frame
• Addresses, households
• Individuals
• At home pattern
• Female employment
• When does the evening start?
• Fieldwork organisation
• Field directors meeting

• Interviewers
• Experience
• F2F
• Random sampling
• Training and briefing
• Remuneration
• Response, hours, staff
• Mode of contacting
• Use of advance letters
• Incentives
• Contact forms

12
Response rates ESS (deviation 70)
13
Why do response rates differ?

• Survey climate
• Used to surveys?
• Popularity topic
• Europe?
• Country size, urbanicity and contactability
• Survey modes
• Predominantly telephone?
• Fieldwork efforts

14

• UP TILL NOW
• NO UNIVOCAL PREDICTOR
• OF
• RESPONSE RATES
• ACROSS COUNTRIES

15
noncontact and refusal (R1, R2, R3)
refusal
noncontact
16
Age, gender, education (Vehovar Zupanic)

• Age underrepresented
• 35 (Austria)
• 55 (Belgium and Luxemburg)
• 15-34 (Spain, Ireland, Netherlands, UK)
• Education underrepresented
• Middle (Austria, France, Iceland, Luxembourg, the
  Netherlands, Norway, Sweden and the United
  Kingdom)
• Less (Switzerland, Czech Republic, Germany,
  Estonia, Hungary, Norway, Slovenia, Slovakia and
  Ukraine)

17
Results multivariate analysis reluctance(Beullen
s, Billiet and Loosveldt)

• Switzerland
• Members of larger families
• Germany
• Female, aged, city dweller
• Internet, unemployment history
• Less political participation
• Estonia
• Female, not in village
• Employed

• Slovakia
• Older, average education, church life
• Never job, less safe, comfortable family income
• Netherlands
• Female, average education, more TV, internet
• Paid job, healthy
• Immigrants threat, trust political institutions
• Civil obedience, political participation
• Socially isolated, dissatisfied

18
Nonresponse bias in cross-national studies

• How to measure bias?
• Different type of auxiliary information available
• Sample frame/register information?
• Reluctant respondents?
• Neighbourhood information?
• Different size of bias?
• Different type of bias?
• Different bias over time?
• ESS-experiments in several countries
• Mixed results

19
Is output harmonisation better?

• Maybe, but in that case do you know about
• Response rates (standard calculation)
• Intensity and effect of field efforts
• Nonresponse composition
• Underrepresented groups
• Nonresponse bias
• Black box
• Differences between countries due to methodology
  or substance?

20
Are comparative surveys possible?

• If not, national surveys are not possible either
• Subgroups
• Socio-demographic
• Survey interest
• Topic relevance
• Regions
• National languages
• Interviewers
• There is so much we dont know, and now we know
  so much more

21
Thank you for your attention