Diapositiva 1

1
The European Conference on Quality in Official
Statistics
Cognitive Test related to Labour Force Survey
ad-Hoc Module on Migrants and their Immediate
Descendants
Mario Albisinni Elisa Marzilli Federica Pintaldi
Rome. 9-11 July 2008
2
The European Conference on Quality in Official
Statistics
Introduction Labour Force Survey

• The Labour Force Survey (LFS) began on
  quarterly basis in 1959
• It is the biggest and most complex sample survey
  - 75.000 households / 170.000 individuals
• - average sampling rate 1/300
• - statistics for small geographical areas (NUTS
  II)
• The LFS is a survey of households that provides
  information about people's employment status and
  conditions
• Official source for employment and unemployment
• Many Eu Regulations

Rome. 9-11 July 2008
3
The European Conference on Quality in Official
Statistics
Introduction Labour Force Survey

• In order to match EUROSTAT standards. on 2002
  ISTAT
• developed a new Labour Force Survey
• It replaced the previous old survey in 2004
• The main changes concern both content and
  technical
• points of view
• - the survey has become continuous
• - from a PAPI technique to a mixed technique
  CAPI-CATI
• - a new questionnaire
• - new information

Since 2005 we have started to diffuse information
about non nationals
Rome. 9-11 July 2008
4
Ad hoc modules specific items every year
The European Conference on Quality in Official
Statistics
Introduction Labour Force Survey
2000 Transition from school to working life
2001 Length and patterns of working time
2002 Employment of disabled people
2003 Lifelong learning
2004 Work organization and working time
2005 Reconciliation between work and family life
2006 Transition from work into retirement
2007 Accidents at work and work-related health
problems
2008 Labour market situation of migrants
Rome. 9-11 July 2008
5
The European Conference on Quality in Official
Statistics
LFS Ad-hoc Module 2008

• The aim is
• complete identification of the population of
  migrants and their immediate descendants
• comparable data at European level
• comparison between migrants (and their
  descendants) and nationals about main
  characteristics of labour market

analysis of the factors affecting integration in
the labour market
Rome. 9-11 July 2008
6
The European Conference on Quality in Official
Statistics
LFS Ad-hoc Module 2008
In order to improve the quality of survey data
for migrants by decreasing language barriers we
conducted a cognitive test
Rome. 9-11 July 2008
7
The European Conference on Quality in Official
Statistics
Cognitive Test

• the cognitive test is a method to reduce
  non-sample errors and improve the quality of the
  data
• the cognitive interview identifies the aspects of
  the question that lead to errors in the answers
• the cognitive interview decomposes and codifies
  the thoughts of the interviewees when they answer
  a questionnaire
• the purpose is to stimulate the respondents to
  suggest the difficulties encountered in answering
  questions

Rome. 9-11 July 2008
8
The European Conference on Quality in Official
Statistics
Cognitive Test

• The cognitive process can be drawn up into 4
  steps
• Comprehension the respondent tries to understand
  the question and its terms
• Retrieval of relevant information from the
  memory the respondent retrieves and selects the
  information in order to be able to answer the
  question
• Decision-making (or judgment) process the
  respondent reflects on what is the correct answer
  and about personal experience to share
• Answer process the respondent decides on the
  answer to be given and communicates it to the
  interviewer

Rome. 9-11 July 2008
9
The European Conference on Quality in Official
Statistics
AM 2008 Cognitive Test

• Methodology
• An interviewer accompanied by one observer
  conducted a face-to-face qualitative interview
• Both the interviewer and the observer had a trace
  they had to follow and which allowed them not to
  go off the point
• Each interview was registered in order to be able
  to listen to it again and transcribe it
• A standardized scheme was compiled to evaluate
  the interviewees level of question comprehension

Rome. 9-11 July 2008
10
The European Conference on Quality in Official
Statistics
Cognitive Test

• Sample
• We interviewed a selected group of 30 migrants by
  sex. citizenship and professional status
• The sample has been divided in two
• - first 15 interviews test questionnaire

some questions were re-formulated because of
problems emerged - second 15 interviews test
new questions
Rome. 9-11 July 2008
11
The European Conference on Quality in Official
Statistics
Cognitive Test
The interview

• The questionnaire contains 3 kind of questions
• Questions to be tested read exactly the way they
  are written on the questionnaire to evaluate
  whether the terms are correct and comprehensible
• Probe questions asked after the questions to be
  tested allow to examine more in-depth the meaning
  of the answers given
• Questions for the observer allow to survey the
  non-verbal reactions of the communication process
  and suggest what kind of information should be
  written on for each question

Moreover the probe questions can be integrated
with other questions the interviewer may freely
elaborate regarding aspects that emerge during
the interview
Rome. 9-11 July 2008
12
The European Conference on Quality in Official
Statistics
Cognitive Test

• The probe questions concern
• problems relative to the question comprehension/
  interpretation
• re-wording of the question by the interviewer
• certainty of the answer given
• strategy used for retrieving past information
• opinion on a specific aspect discovered during
  the interview
• reason for choosing a specific answer and the
  difficulties encountered in formulating it

Rome. 9-11 July 2008
13
The European Conference on Quality in Official
Statistics
Cognitive Test

• The cognitive test questionnaire example

Rome. 9-11 July 2008
14
The European Conference on Quality in Official
Statistics
Cognitive Test
The analysis

• The first step was to transcribe the dialogues of
  the interviews and notes of the observer.
• For each question. we indicated
• answer given
• respondents difficulties
• observers comments

Rome. 9-11 July 2008
15
The European Conference on Quality in Official
Statistics
Cognitive Test
The analysis

• We compiled a standardized scheme to evaluate the
  interviewees level of comprehension
• interviewees question
• answer modality
• pertinence of the answer

Rome. 9-11 July 2008
16
The European Conference on Quality in Official
Statistics
Cognitive Test

• The interviewees question comprehension

Rome. 9-11 July 2008
17
The European Conference on Quality in Official
Statistics
Cognitive Test

• The interviewees question comprehension

Rome. 9-11 July 2008
18
The European Conference on Quality in Official
Statistics
Cognitive Test

• The answer modality

Rome. 9-11 July 2008
19
The European Conference on Quality in Official
Statistics
Cognitive Test

• The pertinence of the question with the
  interviewees condition

Rome. 9-11 July 2008
20
The European Conference on Quality in Official
Statistics
Cognitive Test

• for each question. the combination of the
  information from the content analysis has allowed
  reconstructing a comprehension index
• it was created adding the positive
  comprehension answers and dividing these per
  total number of answers

The index moves from 0 to 100 (max comprehension)

Rome. 9-11 July 2008
21
The European Conference on Quality in Official
Statistics
Cognitive Test

• In relation to the comprehension index of wording
  questions we considered positive comprehension
  answer
• Did the interviewee understand the question
  immediately? Modality yes
• How many difficulty was for interviewee answering
  the question? Modality not much/not at all
• Did the interviewee understand the time
  reference? Modality yes
• Did some words of the question result not very
  comprehensible? Modality no
• Other question comprehension problems? Modality
  no

Rome. 9-11 July 2008
22
The European Conference on Quality in Official
Statistics
Cognitive Test

• In relation to the comprehension index of
  modalities of answer we considered positive
  comprehension answer
• Were the answer modalities clear? Modality yes
• Were the answer modalities exhaustive? Modality
  yes
• How difficulty was for the interviewee have to
  choose an answer modality? Modality not
  much/not at all

Rome. 9-11 July 2008
23
The European Conference on Quality in Official
Statistics
Cognitive Test results
Rome. 9-11 July 2008
24
The European Conference on Quality in Official
Statistics
Cognitive Test results
The new wordings improved the comprehension
levels the average comprehension level of
wording question arises from 71 of the 1st group
to 87 of 2nd group
Rome. 9-11 July 2008
25
The European Conference on Quality in Official
Statistics
Cognitive Test results
The main problems that could arise when using
cognitive test and how they can be corrected
Rome. 9-11 July 2008
26
The European Conference on Quality in Official
Statistics
Cognitive Test some examples
Which is the main reason for coming to Italy?
1_To work 2_To study 3_For
family reasons (reunion. transfer) 4_For
marriage. affective relationship 5_To ask
for political asylum 996_Other
(specify____________) 997_Does not know
PROBLEMS The answer modalities were neither
clear nor exhaustive. The modality to work.
produced some uncertainties for those who came to
Italy to search for a job. R to create myself a
future. I cannot say for work because I did not
have a work contract I came to Italy to find a
job The reason changes in the time R To work.
Well. at first not to stay in Italy. My objective
was to go to Spain and continue to study. Then. I
had to stay in Italy because I did not have much
money anymore I found a job and stayed
Rome. 9-11 July 2008
27
The European Conference on Quality in Official
Statistics
Cognitive Test some examples
Which is the main reason for coming to Italy?
1_To work 2_To study 3_For
family reasons (reunion. transfer) 4_For
marriage. affective relationship 5_To ask
for political asylum 996_Other
(specify____________) 997_Does not know

• CORRECTIONS
• the first modality was divided in look for a
  job and to work
• the verb was put to the past tense to focus the
  attention on the reason for migrating rather than
  on the reasons that subsequently came to create
• a new modality was added short stay in Italy to
  go to another country to survey when Italy is a
  transitory destination rather than the reason for
  migrating

Rome. 9-11 July 2008
28
The European Conference on Quality in Official
Statistics
Cognitive Test some example of results
In the 2 years subsequent to your arrival in
Italy. did you use one of the following services
supplied by public or private institutions/associa
tions 1_ Assistance for searching a job
2_ Professional training (including the one
on the workplace)
PROBLEMS the structure of the question resulted
to be complex and the time reference was not
always understood. I is it clear in the question
that we refer only to first two years? R No. I
understood the whole period

• CORRECTIONS
• the wording of the question was simplified

During the first 2 years you lived in Italy. did
you use one of the following services
Rome. 9-11 July 2008
29
The European Conference on Quality in Official
Statistics
Cognitive Test some example of results
Do you need to know better Italian to find a job
that is more adequate to your skills and/or
previous working experiences? 1_Yes
2_No
PROBLEMS the phrase structure resulted complex
interviewees asked several times to repeat the
question. three terms were not very
comprehensible Italian. skills and
adequate most of the interviewees answered only
the first part of the question (knowledge of the
Italian language) without considering the second
part of the question (work more adequate to ones
own skills and/or previous working experiences)
R what does Italian mean? The Italian
language or Italian as person? R adequate
though is difficult I would say better R to
persons who do not understand well Italian. the
question is too long. At first. one is not sure
of what it is all about R Instead of adequate.
I would say more right for the things you can
do
Rome. 9-11 July 2008
30
The European Conference on Quality in Official
Statistics
Cognitive Test some example of results

• CORRECTIONS
• the question was divided into two parts
• in the first part. respondents are asked if they
  have an adequate job
• if no we asked if the difficulty is because of
  poor knowledge of the Italian language

Do you have a job close to your studies and
competences? 1_Yes 2_No
To have a job close to your studies and
competences. do you need to improve the knowledge
of the Italian language? 1_Yes
2_No
Rome. 9-11 July 2008
31
Ad-hoc Module 2008 Cognitive Test

• Conclusion
• after a second cycle of interviews the
  questionnaire becomes definitive
• the cognitive test has been particularly useful
  in order to set-up of the ahm 2008
• we reduce the non-sample error that bias the
  accuracy of data

Cognitive test highlights hidden errors
Rome. 9-11 July 2008
32
The European Conference on Quality in Official
Statistics
THANK YOU A LOT!
Rome. 9-11 July 2008