CCSA - first results from indepth surveys

1
First results from the in-depth surveys on
quality assurance frameworks and quality
reporting Conference on Data Quality for
International Organisations 27-28 April 2006,
Newport (UK) Håkan Linden, Statistical
Governance, Quality and Evaluation European
Commission, Eurostat
2
Background

• Key areas of work in the Eurostat project on the
  use and convergence of international quality
  assurance frameworks sponsored by the CCSA.
• Problem statement quality assurance frameworks
• - lack of use of quality assurance frameworks in
  most international organisations
• - existence of different quality frameworks and
  benchmarking activities for national statistics
  reported
• Problem statement quality reporting
• - existence of different tools and practices for
  collecting quality metadata from the data
  providers
• - existence of different formats to inform the
  users on the quality of statistics

3
Overview in-depth surveys

• Questionnaire templates developed by the
  task-teams for structuring the work and
  collecting the state-of-art on quality frameworks
  and quality reporting activities.
• All members of the CCSA included in the surveys
• Data collection 20 February 2006 31 March 2006
• 12 international organisations replied thank
  you!
• First results for this conference
• Report with detailed results and recommendations
  to be prepared for the 2nd CCSA meeting 2006

4
Quality frameworks - the template

• Section 6. Quality Assurance Frameworks in place
• - brief description of the quality assurance
  framework
• - legal basis
• - last update
• - coverage of institutional environment,
  statistical processes and outputs
• - quality definition
• - quality requirements/ targets
• - procedures for evaluation of the adherence to
  the quality framework
• - main strengths and weaknesses of the quality
  framework
• Section 7. If a quality framework not yet in
  place
• - brief description of the current situation of
  the quality work
• - how weaknesses are identified, impact assessed
  and improvement actions
• - main obstacles for development and
  implementation of a quality framework
• Section 8. For all international organisations
• - the future plans for quality improvements
• - additional comments/ recommendations etc. for
  this project

5
Quality frameworks - results

• 5 (out of 12) organisations have quality
  assurance frameworks in place
• 1 organisation is currently developing an
  encompassing quality assurance framework
• 6 organisations have not yet begun to develop
  formalised quality assurance frameworks

6
Quality frameworks - in place

• Legal basis
• - endorsed by (internal) Statistical Policy
  Group and Statistical Committee
• - Data Quality Standard under development
• - Article
• - Policy document
• - Adherence to the Principles Governing
  Statistical Activities
• - Recommendation
• Last update 2003 (new update in 2006) May
  2004 July 2003 May 2005 2003.

7
Quality frameworks - in place

• Coverage of institutional environment/ settings
• - all statistical activities/ major activities
  reviewed every 5- years
• - statistical data under direct coordination
  but extension of the coverage planned
• - 2 organisations actually cover the
  institutional settings!

8
Quality frameworks - in place

• Coverage of statistical processes
• - definition of data requirements
• - evaluation of other data currently available
• - statistical design and planning
• - data and metadata collection
• - data and metadata processing
• - compilation and estimation
• - disclosure control
• - data and metadata dissemination

9
Quality frameworks - in place

• Coverage of statistical output
• - all
• - limited to a few domains
• - economic monetary statistics

10
Quality frameworks - in place

• Quality requirements/ targets
• (i) Institutional environment/ settings
• - benchmarks for observation of international
  good practice or standard with reference to for
  example the UN Fundamental Principles of Official
  Statistics, the UN Handbook of Statistical
  Organization, and the Quality Declaration for the
  ESS
• (ii) Statistical processes
• - a set of best practices is defined for each
  process. Each activity must be compared with
  these practices
• - benchmarks
• - standardisation and harmonisation of tools,
  reports and development of policies (like
  revision policy)
• (iii) Statistical outputs
• - a set of best practices is defined for each
  process. Each activity must be compared with
  these practices
• - punctuality (annually agreed timetables)
• - regulations

11
Quality frameworks - in place

• Procedures in place for the evaluation of the
  adherence to the framework
• (i) Institutional environment/ settings
• surveillance missions may help to identify
  institutional and legal issues underlying
  problems in data quality
• audit exercises
• self-assessments
• peer reviews
• annual report on the implementation and
  protection of confidential information
• (ii) Statistical processes
• - quality reviews of ongoing activities on a 5
  year rolling basis (self-evaluation of strengths
  and weaknesses by comparing with best-practices)
• (iii) Statistical outputs
• - compliance reports
• - regular reports on gaps in statistics (as
  input for medium-term work-pgm)
• - quality reports/ quality profiles

12
Quality frameworks - in place

• Strengths
• - Quality reviews (self-evaluation of strengths
  and weaknesses by responsible persons, comparing
  their activities with best-practices) on a
  rolling basis give a good framework for
  discussing quality problems between domain
  managers and statistics and IT- experts
• - Data quality monitoring at each stage of a
  statistical production process makes it possible
  to identify and address data quality problems.
  Incorporated feed-back loops serve as a
  mechanism for improving data quality
• - The implementation of quality framework
  (phased/ staggered approach) ease the development
  of new work processes related to data collection
  and processing, IT, and data dissemination
• - Existence of transparent and comprehensive
  legislative framework for the production of
  statistics ease the monitoring of data quality
• - Quality manager appointed to coordinate all
  quality work

13
Quality frameworks in place

• Weaknesses
• - The resources allocated to corporate quality
  work are insufficient
• - The resources to help solve quality problems
  are scarce
• - The implementation of process-oriented data
  quality management requires strong co-ordination
  and support (IT and statistics)
• - Lack of standardisation of data treatment
  procedures and insufficient documentation of
  methods

14
Quality frameworks in place

• The main constraints on reaching optimal data
  quality in the statistical processes
• - The resources
• - Difficult to apply the same quality concept
  (e.g. dimensions) for less developed statistical
  systems as for more developed systems
• - The improvement of data quality is viewed as a
  gradual process that need to take into account
  resources constraints and establish priorities
• - No direct contact with respondents
• - The quality of the country data

15
Quality frameworks - not (yet) in place

• The current quality work
• - implementation of new Statistical Information
  System (data validation/ consistency checks and
  metadata management)
• - procedures for validation of statistics/
  estimations by countries before publication
• - regular review of data collection activities
• - internal quality reports
• - methodological aspects are documented and used
  to identify best practices
• - project on process documentation to identify
  weaknesses and define best practices

16
Quality frameworks - not (yet) in place

• Identification of weaknesses
• - manual data clearance procedures are well
  functioning but proves to be time consuming and
  intervenes at a late stage in report preparation
• - data weaknesses are identified when in-house
  economists draft their analytical reports based
  on the statistics
• - data weaknesses are identified on the basis of
  metadata available and by comparisons with
  similar statistics from other organisations
• - the impact of data weaknesses is assessed
  using statistical judgement, improvement
  actions are discussed between statistician(s) and
  the supervisor and implemented according to
  agreed plans
• - weaknesses in data are pointed out by the
  users (user feed-back or user surveys)

17
Quality frameworks - not (yet) in place

• Main obstacles/ critical issues for
  implementation of quality assurance framework
• - economists and analysts perceive data quality
  requirements too excessive and seen as extreme
  refinements imposed by statisticians
• - relevant IT tools should support the quality
  framework to efficient process of data and
  quality control (QF might be seen as a concept
  impossible to apply)
• time and resources
• A well-designed quality framework for
  organisations compiling statistics from different
  international organisations would ease the
  implementation

18
Quality frameworks future plans (1)
- review of existing quality framework -
inform senior managers on data quality issues and
principles governing international statistics -
training of staff on quality issues and data
processing - data collection by electronic
means will increase and make it possible to
implement more automatic data validation - SDMX
is expected to be a mean for accessing data and
metadata from countries and other international
organisations - improved accessibility to data
for users - adapt the common quality assurance
framework that will be proposed by CCSA -
establishment of a statistical data quality group
19
Quality frameworks future plans (2)
- promoting data and metadata dissemination -
regular monitoring, assessment and reporting
based on the statistical data quality
framework - peer reviews of member countries
20
Quality reporting the template

• Section 6. Metadata about quality collected from
  data providers
• - brief description of the quality reporting
  activties
• - legal basis
• - quality dimensions covered
• - frequency
• - procedures/ mechanisms in place for quailty
  reporting
• - type of quality information (qualitative/
  quantitative)
• - the use of the quality information
• - strengths and weaknesses of the quality
  reporting
• - quality constraints
• - quailty improvements
• Section 7. Metadata about the quality of released
  statistics
• - brief description of the ways of informing the
  users on the quality
• - based on standardised reference metadata
• - type of quality information disseminated
  (qualitative, quantitative)
• - strengths and weaknesses of the system of
  informing users on quality
• Section 8. For all international organisations

21
Quality reporting - results

• Legal basis
• - Guidelines for good practices
• - Ratification of Convention for data reporting
  and its implementation
• - indirect through Council decision
• - Standards
• - Regulations
• - Recommendation

22
Quality reporting - results

• Coverage of quality dimensions (1)
• - relevance, accuracy, credibility, timeliness,
  accessibility, interpretability, coherence
• - relevance, accuracy, timeliness, punctuality,
  accessibility, clarity, comparbility, coherence,
  completeness, and sound metadata
• - prerequisites of quality, assurances of
  integrity, methological soundness, accuracy and
  reliability, servicability, accessibility
• - methodological soundness, accuracy,
  reliability, consistency, timeliness, and
  punctuality
• - relevance, accuracy, timeliness, punctuality,
  accessibility, clarity, comparability, and
  coherence

23
Quality reporting - results

• Coverage of quality dimensions (2)
• - relevance, accuracy, interpretability, and
  coherence
• - completeness, accuracy, reliability,
  comparability, adherence to standards
• - data accuracy, comparability and consistency
• - data consistency (aggregation) and
  comparability with standards
• - methodological soundness (coverage) and
  conistency over time

24
Quality reporting - results

• Frequency of the quality reporting
• - no common rules
• - rolling programme
• - once a year
• - approx. every 5 years
• - twice a year
• - each time data submitted/ collected
• - quarterly
• - according to requirements in Regulations
• - as part of projects on country assistance/
  country reviews

25
Quality reporting - results

• Procedures/ mechanisms in place for the quality
  reporting
• - the quality framework provides the theoretical
  and practical guidance
• - metadata questionnaire attached to statistical
  data questionnaire
• - manuals/ guidelines/ handbooks
• - clerical and computerised edits
• - standardised reporting forms for metadata
• - glossary on quality
• - bulletin board on data collection,
  dissmeination and quality of statistics
• - standardised data quality assessment framework
  tool (part of formalised data quality progam) for
  assessing member countries data quality
• - regulations
• - metadata common vocabulary

26
Quality reporting - results

• Type of quality information collected
• - almost all information provided are
  qualitative
• - very few quantitative measures, like
• - response rates
• - data completeness
• - data timeliness
• - revisions
• -------------------------
• - binary indicators
• - set of standard quality indicators
• - overall assessments
• - allocation of colour-codes to country data
• - quality ratings (four grade scale)

27
Quality reporting - results

• Strengths
• - decentralised meaning that activity managers
  can collect the metadata
• - metadata reporting by NSOs are done in a
  systematic manner
• - allows checking compliance against
  international standards
• - a tool for monitoring where there are
  trade-offs between data quality problems and
  allocation of resources
• - standard framework for metadata in general and
  metadata about quality in place covering many
  countries
• - good cooperation with data providers
• - quality criteria and quality reporting
  requirements included in legislations for member
  countries

28
Quality reporting - results

• Weaknesses
• - un-coordinated
• - too decentralised (no coordination)
• - no standards/ standardised reporting
• - no central monitoring
• - needs further embedding in overall culture of
  the organisation
• - reporting NSOs not aware of international
  standard methodology, concepts, definitions and
  classifications
• - no standardisation of metadata and rules to be
  followed
• - metadata are not processed automatically
• - infrequent collection of information
• - burden on countries
• - lack of resources
• - self-assessments of countries not yet done
• - not all statistics yet covered

29
Quality reporting - results

• Coverage of quality constraints
• - none (9 out of 12)
• - data sharing and coordination among data
  producing agencies
• - confidentiality
• - resources (financial, staff, technical) and
  efficient use
• - planning of statistical programs
• - covered by a special merit and costs
  procedure
• - burden on respondents

30
Quality reporting - results

• Coverage of quality improvement actions
• - no (8 out of 12)
• - sometimes
• - explicit sections on plans for improvement
  and specific recommendations for improvement

31
Quality reporting - results

• Future plans for improving quality reporting from
  data providers
• - continued use of electronic dissemination
• - investigation on the use of SDMX standards
• - development of a metadatabase that allows
  automatic handling of metadata for the
  processing, tabulation and dissemination
• - revision of the quality reporting framework in
  order to improve standardisation
• - data quality assessment framework is being
  incorporated into the data dissemination
  standards
• - development of a coherent quality framework
  with the aim to introduce a harmonised definition
  of quality
• - workshops and seminars for countries on
  strengthening the capacity to produce data

32
Quality information for the users

• Ways of informing the users on the quality
• - quality metadata is embedded in the
  dissemination system
• - attached notes to datasets and tables
• - metadata included in Sources and Methods
  publications
• - standardised country notes on methods and
  definitions for each release
• - Data Dissemination Standard(s)
• - dissemination of data only possible if
  metadata are filled in
• - annual quality report published summarising
  all economic statistics
• - annual internal quality report for each
  statistical area
• - quality profiles for key statistical
  indicators
• - domain specific quality reports

33
Quality information for the users

• Future plans for improving quality information
  about the published statistics
• - implementation of new Statistical Information
  System designed to fully support metadata
  (harmonisation and automatisation)
• - more use of pre-coded response for metadata to
  improve standardisation
• - adoption of systematic criteria is under
  consideration (dependent on international work)
• - coverage of all statistical domains
• - data quality requirements to be better
  integrated in the statistical database
  dissemination system