Recording administration of pilot questionnaires:

1
Recordingadministration of pilot questionnaires

• An efficient method to improve
• the quality of data
• Géraldine Vivier
• INED (National institute of demographic studies)
  Paris
• Surveys Department
• Cardiff, 2006

2
Questionnaire construction and data quality

• To test questionnaire phrasing, improve its
  quality by reducing possible misunderstandings
  between interviewers (questionnaire) and
  interviewees
• Different methods
• Split sample comparisons
• Cognitive interviews
• Interviewer debriefings
• Recordings of pilot questionnaire administrations

3
Recording pilot questionnairesObjectives and
main interest 

• Capturing what is said about or around the
  questionnaire
• And how it is said.
• It enables us to catch elements concerning
• Interviewees understanding of questionnaire
• Meaning of interviewees responses
• Line of thought, mental processes and means
  required to answer
• That is elements about
• the cognitive process of responding

4
What is picked up through recording? Examples
from a French life event history surveyconducted
among people aged 50 to 70

• Ambiguities misunderstandings intentionally
  expressed or not by interviewees
• 1. Questions or Precisions asked by
  interviewees
• - Are currently you in a relationship?
• - Do you mean do I live with someone or do I have
  a partner?
• 2. Explanations given unintentionally
• - What years were your parents born in? Father
  Mother
• - My mother was born well, my adoptive mother,
  well so to speak because they didnt adopt me
  formally but since my biological parents died
  during the war, I was taken in by the neighbours
  they took care of me and became my parents. So my
  adoptive mother, she was born in

5
What is picked up through recording?

• ?? Recording shows potentially invisible
  ambiguities between a large scope of links which
  can compose parenthood biological,
  institutional, factual and elective links...
• ?? Rephrasing the question to specify more
  exactly what is expected
• ? 21 of interviewees quoted a person as a
  parental figure alternative or in addition to
  their biological (or adoptive) parents.

6
What is picked up through recording?

• Responding processes and bias linked to interview
  situation,
• Norms and interviewer-interviewee relationship
• - How many brothers and sisters do you have?
  Brothers Sisters
• - Well for you, I would say 7, 5 brothers and 2
  sisters.
• Simplification, conformism, desirability effects

7
What is picked up through recording?

• Interviewees reactions,
• Interviewees words and phrasings
• Did you cohabitate beforehand getting married?
• Oh no! no! Not at all! My goodness no! Youre
  young but you know, in those days for our
  generations, cohabitation wasnt the done thing!!
• But later, she explains that she set up house
  with her second partner a few months before
  getting married to him
• ? Questions rephrasing using interviewees
  words.
• ? Finally 23 of all married unions (regardless
  of ranks) were preceded by cohabitation 56 of
  2nd marriage 19 of 1st ones!

8
What is picked up through recording?

• Other elements about interviewer-interviewee
  relationship, interviewers work and practices
• - Indications about interviewers embarrassment,
  uncomfortable situations Tone of voice or
  questioning rhythm changes
• Interviewers bias errors in questioning,
  rephrasing
•   ? To (again) improve the questionnaire
• ? To prepare interviewer debriefing
• ? To improve interviewer training

9
In fact, recordings enable

• to identify difficulties, including
  interviewers in order to
• Precise, clarify or adapt concepts, categories,
• Take into account unexpected dimensions
• Improve phrasing, vocabulary
• Reduce misunderstandings
• Improve the situation of interview,
  interviewer-interviewee interactions
• Prepare and perform better interviewer trainings

10
Advantages Limits of the method Discussion (1)

• Can be undertaken for each session of
  questionnaire testing
• No transcription needed listening with
  note-taking is sufficient
• ? A time and money saving method
• No specific skills required from interviewers
• No specific questions about the questionnaire
  content or interviewees thinking and
  understanding
• Observation of the questionnaire administration
  without any intervention
• ? An efficient and simple method

11
Advantages Limits of the method Discussion (2)

• A non systematic process Identification of
  difficulties partly depends on what is
  spontaneously expressed by interviewees
  (including silences, laughs) What is thought is
  not necessarily said aloud
• Difficulties, reactions and behaviours of
  interviewers are also picked up.
• ? A complementary method in addition to
  interviewer debriefings, and other methods used
  to construct and test questionnaires

12

• THANK YOU!