A reference architecture for automatic XML data

1
A reference architecture for automatic XML data
metadata exchange between public administrations
Eurostat's case study

• Dr. Kyriakos Kassis
• Agilis SA Statistics and Informatics
• GREECE

2
Eurostat and the European Statistical System /
ESS

• Network of national institutions and agencies
• Co-ordinated by Eurostat
• European Commission Directorate General
• European Union Statistical Authority
• Tasked with providing a high quality statistical
  information service
• 1000 data providers to Eurostat
• More than 100 committees, working groups and
  expert groups

3
Harmonisation methodology coordination

• EU Regulations and Agreements
• Code lists, nomenclatures
• Reporting tools, services and standards
• National raw data
• Aggregated European data
• Latest technologies
• Internet, Web services, XML
• SDMX

4
Aim and objectives

• Implement a modern data exchange, sharing and
  dissemination scenario for both data and metadata
  in the ESS facing the challenge of XML, web
  services and web feeds
• Implementation of specific contracts
• SODI SDXM Open Data Interchange
• SDMX registry
• SDMX Reference Metadata
• financed by IDABC - the European Commission
  program on interoperability
• Act as the primary vehicle for driving forward
  the whole range of technical, organisational,
  statistical, legal, and political developments
  involved in implementing SDMX in the ESS

5
Data Sharing approach

• National Statistical Institute (NSI)

Collection
Production
Dissemination
shared data
Collection
Dissemination
Production
Eurostat
6
Interoperability in the ESS

• Interoperability integral part in ESS
• Need for high quality, timely statistical data
  with well defined semantics between the figures
• Key players in modern data exchange scenario
• International organisations (MS, UN, OECD, IMF
  ECB etc.)
• Business users
• Citizens
• Facts
• Act simultaneously as data providers and data
  consumers of statistical information
• Automatically extract statistical data described
  using the same metadata during the process of
  collection and transmission

7
Architectural Guidelines

• Web services technology
• Non proprietary solutions and software (OSS
  tools/libraries, Java)
• Data and Metadata transmission through the pull
  push approach
• Ensure cooperation with already existing
  Eurostats operating environment
• eDAMIS/SEP
• Databases Eurobase, EMIS
• Widely used standards and related technologies
• SDMX version 2.0 standard for query messages
  datasets
• Web Feeds (ATOM 1.0 and RSS 2.0)
• W3C standards (XML Schema, XSLT)
• SOAP
• Web Services Interoperability Organisation

8
Statistical Data and Metadata Exchange (SDMX)

• International initiative to improve data and
  metadata exchange and sharing
• BIS, ECB, Eurostat, IMF, OECD, UNSD, WB
• Version 1.0 is ISO TS 17369, version 2.0
  submitted
• Content-oriented guidelines drafted
• Pilot projects have demonstrated technical
  feasibility
• Memorandum of Understanding on organisation
  issues agreed by sponsors in Jan 2007
• Implementation strategy in the European
  Statistical System was endorsed by Statistical
  Programme Committee in Mar 2007

9
Pull scenarios and web feeds

• Static Scenario
• Data provider (NSI) prepares new or updated data
  in SDMX-ML format and places them in specific URL
  (http, ftp, data delivery space of a WS)
• Subscribers are notified by a simple feed on the
  availability and location (URI) of these data and
  fetch them
• Requires only the URI of the published dataset!
• Dynamic Scenario
• NSI generate web feed notifications which contain
  the description of the new or updated data in
  terms of SDMX
• Subscribers parse this notification and act
  accordingly (submit the query for the entire
  dataset or restrict it)
• Requires apart from the location structural and
  reference metadata of the published dataset !

10
SODI Architecture
NSI
Web Feed
Pull requestor
SDMX Registry WS
Web Service WS
SDMX Web Service
SDMX Publisher
SODI Processing Environment
11
SDMX Registry

• A repository for maintaining and providing
  structural and reference metadata
• Code Lists
• Concepts, Concept schemes
• Data Structure Definitions (DSD)
• Metadata Structure Definitions (MSD)
• Repository is accessible via a Web Service
  accepting SDMX-ML messages
• GUI for user interaction over the Web
• Supports SDMX version 2.0
• Authentication using CIRCA accounts

12
(No Transcript)
13
Component Diagram
14
Deployment in Eurostat
15
Future implementation in the ESS

• Actions on data and structural metadata
• Improving data transmission features
• Bridging gap between reception and dissemination
• Improving dissemination of data and metadata
• Domain-specific SDMX-compliant data descriptions
• Assistance for Member States
• Actions on reference metadata and content issues
• Use of a common and consistent terminology
• Promotion of common metadata concepts
• Common platform for reference metadata exchange
  in the EU

16
Conclusions

• Strong commitment of Eurostat to SDMX
  implementation
• Pilot implementation for the PEEIs
• Extension to other potential statistical domains
  demography, education, balance of payments
• Face opportunities and challenges related to new
  and emerging technologies
• Investement on resources and technological
  environment is kept at minimum level
• SODI, SDMX registry and SDMX Reference Metadata
  as driving projects
• Ensure continuity with the past and legacy
  systems
• Feedback to SDMX standard development