European Conference on Quality in Survey Statistics

1
European Conference on Quality in Survey
Statistics
2
Legislation, Codes of Practice and the Quality
of Official Statistics

• Len CookNew Zealand

3
The role for society of the statistician

• "The statistician keeps his fingers on the pulse
  of humanity, and gives the necessary warning when
  things are not as they should be."
• Quetelet

4
The place of official statistics in timely and
decisive action
?
Official statistics
Research
Theory
Case studies
Experience
Policy Framework
Anecdote
Myth
Stereotype
5
How much is confidence in official statistics in
a state of change?

• Increased expectations
• Changing World
• Greater use
• Privacy
• Quality demands
• Technology
• Accessibility
• Advances in methods and practice

6
Prospects and trade-offs for the Global
statistical system
Competitive influential Being able to be
decisive and timely in actions
Prospects
Sense of Place of Official Statistics
Delivery Performance
Trade-offs risks in reputation quality
Trade-offs
7
Summary of key elements of statistical
legislation
Unlikely to be in legislation Strengthened by legislation Usually in statistical legislation
Quality standards Roles of Head of National Statistical Institute, and responsible Minister Authority to obtain individual information
Relevance Accessibility Protection of individual responses from disclosure
Responsiveness Managing respondent load Objectivity of methods
Government/ community balance Co-ordination across government (survey approval) Impartiality of results, and selection of measures
Efficiency Definition of official statistics Capacity to link records
8
Responsiveness of statistics and the nature of
theprofessional/political boundary
x service agreement (key users)
x political consensus
x structured user engagement/interaction
x community partnerships (ethnicity)
RESPONSIVENESS
x manage obsolescence (resources)
x benchmarking, reviews
x performance targets
x quality framework
x other legislation
x political oversight
x international standards
x comprehensive statistical legislation
x codes of practice
POLITICAL
PROFESSIONAL
9
The diminishing capacity to make quality
assessments
x service agreement (key users)
x political consensus
x structured user engagement/interaction
QUALITY INFORMATION
x community partnerships (ethnicity)
x manage obsolescence (resources)
x benchmarking, reviews
x performance targets
x quality framework
x other legislation
x political oversight
x international standards
QUALITY MEASURES
x comprehensive statistical legislation
x codes of practice
POLITICAL
PROFESSIONAL
10
The gap between capacity to measure quality and
expectations
x service agreement (key users)
x political consensus
x structured user engagement/interaction
QUALITY INFORMATION
x community partnerships (ethnicity)
x manage obsolescence (resources)
USER EXPECTATIONS OF QUALITY
Knowledge gaps
x benchmarking, reviews
x performance targets
x quality framework
x other legislation
x political oversight
x international standards
QUALITY MEASURES
x comprehensive statistical legislation
x codes of practice
POLITICAL
PROFESSIONAL
11
The growing gap between expectations and quality
measures
QUALITY INFORMATION
Knowledge gaps
USER EXPECTATIONS OF QUALITY
QUALITY MEASURES
POLITICAL
PROFESSIONAL
12
Legislation and quality
x service agreement (key users)
Statistics released without being undermined
Objective selection of measures Access to
administrative records Increased record
linkage Strong survey response rates Continually
robust survey frames Widen unit record access for
research Expand small area statistics Reduce risk
of accidental disclosure
x political consensus
x structured user engagement/interaction
x community partnerships (ethnicity)
x manage obsolescence (resources)
x benchmarking, reviews
x performance targets
x quality framework
x other legislation
x political oversight
x international standards

Authority to act Disclosure protection Objectivity
of methods Impartiality of results
x comprehensive statistical legislation
x codes of practice
POLITICAL
PROFESSIONAL
13
Parliament is the place to recognise when
ministerial authority has been transferred

• Ministers are elected to Parliament to make
  decisions for their peoples. If the authority
  to act in one sphere of responsibility, official
  statistics, is to be constrained, it should be
  recognised by Parliament, in the legislation of
  Parliament. Without this recognition, deciding
  what accountability will remain with Ministers
  may lack consensus, clarity, or consistency.
• Often statistical legislation will give a clear
  delineation in the responsibilities of the
  countrys chief statistician and the respective
  minister in the government responsible for it.
  This explicit relationship can be taken as that
  which applies to all Ministers and to all
  official statisticians.
• Comprehensive statistical legislation reduces
  expectation that each statistical decision or
  statistical event can be managed to have a good
  political outcome.

14
The place of legislation in the protection of
confidentiality

• Obligation
• International security
• Protecting archives
• Data-matching and increased register use
• Challenging perceptions
• Quality of protection
• Increased capacity to access records from
• Citizens (high response rates)
• Business (tax records and market sensitive
  information)
• Government (administrative records)
• Capacity to protect (court and police requests)
• Limit power of situational ethics
• Limit spread of less relevant oversight

15
The place of legislation in ensuring the
objectivity of methods

• Selection of methodology by statistician
• Highly political interventions should be by
  legislation or judicial process
• Fitness for use in unknown situations outside
  design criteria
• International coherence
• Open peer review of practice
• Access to unit record data
• Immediacy of linkage to politically sensitive
  outcomes places a greater constraint on the
  reflective, consultative processes of the
  statistician
• Need high standing of role of National
  Statistician
• High rate of obsolescence of methods and
  conceptual reorientation
• External drivers grown tax liability, benefit
  eligibility, public funding, EU comparability,
  international quota
• Growth in international frameworks,
• Demand for cohesiveness across independently
  managed statistical sources
• Emphasis on residuals and balances from large
  gross flows
• Performance targets selected by Ministers

16
The place of legislation in impartiality of
results and practices

• publish information about quality without
  approval
• independent publication of compendia on any topic
• have sufficient finance for assessing quality
• judgement on early access and immediacy of policy
  statements
• choice of comparisons
• obligation of consistency
• respondent relevant description and concepts
• Indigenous recognition, language use
• No intervention in senior appointments
• Limit damage caused by early access
• Limit juxtaposition of statistical release and
  policy reaction
• Independence of selection of comparisons

17
The place of legislation in impartiality of
results and practices (contd)

• Release of information about the quality of
  results can affect confidence in policies based
  on them
• Timing of release and quality standards brings
  tensions that can result in belief in political
  intervention
• Need to publish methods and available evidence
  used in judgements even if confidential for
  other reasons
• Reliance on quality not being a sensitive concern

18
Can legislation advance quality, given that it is
a matter of culture and method?

• Quality focus in statistical offices sometimes
  drives major strategies - statistical
  modernisation (UK)
• But mostly where staff have a quality focus they
  make sure that
• Innovation often results from quality failure
• Value added by users does not damage the
  contributing products
• Avoidable limitations to quality are fixed

19
Innovation often results from quality failure

• Unanticipated revisions
• Delay in applying benchmark series revisions to
  rebased series
• User enquiries cannot be answered
• Users find errors
• Users get inconsistent answers on different
  occasions
• Derived measures use sources inconsistently
• Users find inconsistencies across series that
  require revisions
• Users fail to receive statistics as ordered
• Instability and inconsistency from results of
  related series from different sources
• Known policy applications delayed for statistical
  faults

20
Value added by users may reduce trust in NSI

• Users place statistical results into standard
  databases
• Users add metadata to series
• Users add to coherence with related series
• Users add explanations of series volatility
• Users need to check unusual results

21
Avoidable limitations to quality get fixed

• Inadequate range of quality measures
• Expected revisions delayed
• Insufficient length of series
• UK or only some of England, Wales and NI
• Inconsistent results across related series from
  different sources
• Inexplicable delays in getting results

22
What is essential alongside statistical
legislation?

• Strong formal user involvement
• Rich publishing culture
• High degree of analysis, including complex
  derived statistics such as productivity, poverty
  and fiscal incidence
• Access to unit record datasets
• Large number of processes for providing
  transparency
• Regular publication of measures of cohesiveness/
  confrontation across statistical sources
• Strong peer review processes
• Regular cross country benchmarking activity
• High degree of linkages across the collection and
  integrating frameworks produced by the system
  (Nat accounts, demography)
• Parliamentary ownership of system
• Strong methodology teams
• Engagement in cross country developments of
  statistical frameworks and standards
• International exchange of staff from both
  professional and operational areas of statistics

23
Codes of Practice and quality frameworks

• We may be overstating our capacity to measure
  quality
• We may not have sufficient resources to
  systematically assess quality through a
  comprehensive quality framework
• Hard to measure statistics that are very
  important may never be assessed
• We could give more attention to assessing the
  measurability of key phenomena, where measures
  greatly influence public life
• We may be undervaluing our judgements about
  quality in difficult uses of statistics by giving
  emphasis to tangible quality measures

24
Conclusion

• Legislation plays a key role in quality, but
  cannot substitute for user engagement,
  transparency and judgements on measurability.
• Legislation affirms the fundamental obligations
  on the statistician, and the authority to obtain
  information that that is central to the job
• The coming decades will see great change in our
  capacity to retain confidence in the quality of
  our work.
• We will give more confidence through the
  certainty of our processes
• The accessibility of statistical sources will be
  the most significant source of innovation, once
  technological transformations have been achieved

25
European Conference on Quality in Survey
Statistics