An integrated information system on survey quality: the experience of the Italian survey Holidays an

1
An integrated information system on survey
quality the experience of the Italian survey
Holidays and trips
7th International Forum on Tourism Statistics
For Usability, Comparability and
Sustainability Stockholm, Sweden 9-11 June 2004

• by Monica Perez

Authors Perez M., Head of Unit Mobility and
tourism, Division for Survey on Households
Dattilo B., Statistician, Unit Mobility and
tourism, Division for Survey on
Households Simeoni G., Statistician, Unit
Methodologies and standard for quality, Division
for Methodologies and Information
Technologies
Italian National Statistical Institute
2
Table of contents

• The survey Holidays and trips (HT) its
  context, framework and main features
• The monitoring system on survey quality adopted
  for HT
• The Information System for Survey Documentation
  (SIDI)
• SIDI quality indicators
• How SIDI quality indicators have been implemented
  in HT
• Ad hoc quality indicators adopted by HT

3
The HT main features

• HT is a quarterly survey carried out on a
  household sample in order to comply to Section C
  (demand side) of Directive 95/57/EU
• HT is mainly aimed at getting info on overnight
  stays (personal/business), domestics and outbound
  tourism flows, tourist patterns, etc
• The sample
• size 14,000 households (3,500 per quarter)
• design stratified by 20 regions and 6
  socio-demographic typologies of municipality
• The frame is a national (sub) list of Telecom
  companys subscribers (private users)
• The survey is carried out by Computer Assisted
  Telephone Interviews (CATI)

4
HT monitoring needs

• The results can be affected by non-sampling
  errors ? the possibility to have info on data
  collection - survey contents and how the survey
  is proceeding- provides an added value to
  understand the investigated phenomena
• Survey field period 20 days every 3 months ?
  timeliness in detecting problems arising during
  the data collection phase
• Interviews carried out by a private firm
  ?strengthen control activity on interviews phase
  and monitoring interviewers performance

5
The monitoring system on survey quality adopted
for Holidays and trips (HT)
4 monitoring activities
monitoring activity carried out by Istat
specialized staff directly at the office of the
firm in
charge to carry out the interviews
monitoring activity carried out by control
telephone calls to the
interviewed households made by
Istat staff
monitoring activity carried out
by the analysis of a set of
statistical indicators and
reports on interviewers performance and data
collection for the
qualitative
analysis
monitoring activity carried out by a toll free
telephone number
working during
the whole survey period
6
Reporting and statistical indicators daily
provided for HT quality monitoring system

• data collection reporting
• Tools statistical frequency distributions (a.v.
  and ) on each variable
• call outcomes rates reporting
• Tools a set of statistical indicators on survey
  performance
• data files on call outcomes
• Tools
• Call final outcome (CFO) a database containing
  info related to telephone calls that have
  provided a final outcome
• Call provisional outcome (CPO) a database
  containing info on each call outcome (calls with
  a final outcome and calls in a temporary status)

7
The Information System for Survey Documentation
SIDI

• SIDI is the Istat centralised system for survey
  documentation and quality assessment
• Allowing comparisons on the quality of Istat
  surveys
• Purpose supporting survey managers in quality
  control activities
• Monitor the production process and quality over
  time
• Main features
• Process-oriented system
• Integrated management system of metadata and
  quality indicators
• Architecture two subsystems
• SIDI1 management subsystem developed in Oracle
  forms
• SIDI-TOP navigation subsystem developed in Java
  available on Istat intranet

8
SIDI contents

• Metadata available for each survey
• Information contents (observed phenomena,
  analysis units)
• Production process description (operations e.g.
  data collection mode - quality control actions
  e.g. interviewers monitoring -)
• Standard quality indicators, a set for each of
  the most important survey phases
• Frame Relevant for the monitoring
• Data Collection of data collection phase
• Data Entry
• Editing Imputation
• Timeliness Punctuality
• Costs
• Coherence

9
SIDI Standard quality indicators

• Main features
• Standard formulae, calculated by the system
  survey managers provide numerators and
  denominators
• Tabular and graphical representations of the
  indicators provided by SIDI-TOP
• Over time monitoring and geographical analysis
• Comparisons among different surveys
• Comparisons with average values
• Implementation for Holidays and Trips
• First CATI survey -using the phone book as a
  frame- that has calculated SIDI indicators on
  frame and data collection
• adaptation of SIDI general definitions to
  specific survey situation

10
How SIDI quality indicators have been implemented
in HT

• SIDI units classification has been revised and
  improved to better fit HTs needs and those of
  similar surveys as well
• Operative definitions of SIDI quantities to
  calculate indicators have been provided
• A re-allocation of some call outcomes with
  reference to SIDI categories
• Basic quantities to calculate SIDI indicators
  have been identified and processed by call
  outcomes data files (CFO CPO)
• SIDI indicators have been implemented in the
  daily reporting for HT quality monitoring

11
An overview of SIDI indicators
12
(No Transcript)
13
Response Rate. Years 2001, 2002
14
Refusal Rate. Years 2001, 2002
15
SIDI
Response rate by geographical breakdown
HT
16
Ad hoc quality indicators adopted by HT

• Additional indicators not supplied by SIDI are
  recommended to fully satisfy each quality
  monitoring needs
• The core set of indicators is broken down by
• Interviewer
• Geographical region
• Timing of interviews (timetable and duration by
  final status code)
• Call outcomes sequence
• Opinions on interview (general difficulty,
  difficulty by questionnaires section, refusal
  reason, info on refusal person, )
• Info on specific aspects on collected data ( of
  travelling persons/households, missing values on
  travel expenses, etc.)

17
Closing comments

• Monitoring the survey is an important chance that
  should be taken into consideration by researchers
  and survey managers
• Having info on data collection during and after
  the data collection phase is recommended because
  it is an added value to understand the
  investigated phenomena
• Using a quality management system is a first
  important step to understand how much non
  sampling errors can affect the final survey
  results

18
Thank you for your attention !