Earth Observation Payload Data Long Term Archiving The ESA MultiMission Facility Infrastructure

1
Earth Observation Payload Data Long Term
ArchivingThe ESA Multi-Mission Facility
Infrastructure

• Gian Maria Pinna ESA
• Eberhard Mikusch, Manfred Bollner DLR
• Bernard Pruin Werum Software Systems AG
• PV-2005
• 21-23 November 2005

2
MultiMission Facility Infrastructure
(MMFI) Rationale

• In 2003 the Agency approved a strategy for the
  evolution of the several missions ground segments
  (handled and/or to be developed) into an open
  multi-mission architecture, in accordance with
  the Oxygen concept for harmonized ESA E.O.
  Ground Segment implementation
• Basic implementation principles
• Adoption of a common architecture for all
  missions for generic ground segment, mission
  independent
• Decomposition of the facility architecture into
  functional block elements
• Identification of mission specific and common
  elements
• Harmonization and standardization of interfaces
• Maximum re-use of already proven elements
• Standardization of products and formats across
  missions
• Adoption of strategy for all future ESA handled
  missions
• Extension of concept to European National
  missions
• Harmonization and rationalization of archives

3
Payload Data Ground Segment Decomposition
Specific-to-mission elements(Processors,
Acquisition, Q/C, etc.)
Examples of multimission common elements
4
Payload Data Ground Segment Logical
Model (OAIS-based)
5
PDGS Services Model

• A PDGS for a generic mission is composed of
• a Multi-Mission Central Infrastructure component,
  consisting of all elements required to provide
  User Services (cataloguing, user access, data
  ordering, etc.), and Quality Assurance services
  (payload data quality control, sensor performance
  assessment, etc.)
• a distributed Multi-Mission Facility Ground
  Segment (FGS) component, consisting of all
  elements necessary for the acquisition,
  ingestion, long-term archive, order processing
  and data disseminations to end users.

6
FEOMI

• The project FEOMI (Facility Evolution into an
  Open Multimission Infrastructure) aims at porting
  the ESAs FGSs into an MMFI-based architecture,
  by
• Analyzing the present existing operational
  requirements
• Consolidating the MMFI architecture in order to
  satisfy all identified operational requirements
• Implementing the specific configuration in the
  Core MMFI to support all operational missions
• Developing new or Adopting/Adapting existing
  elements to build the new MMFI, in line with the
  basic concepts and logical model
• Developing a generic infrastructure for the usage
  of the MMFI by future missions FGS

7
FGS Logical Model Definition

• The FEOMI Requirements Analysis phase highlighted
  the need for
• explicit interoperability model elements for data
  and metadata exchange, to take care of the
  dynamics of the distributed archive with
  federation and co-operation between sites for
  data exchange, and the synchronization with the
  central catalogue for metadata and browse images
• the mapping of the model elements onto components
  that were either already in operations in the ESA
  FGSs, could be created by configuration of COTS
  or required to be custom built

8
Facility Ground Segment Logical Model
Multi
-
Mission Central Infrastructure
Other FGS
Multi
-
Mission Central Infrastructure
Production
Monitoring
Production
Exchange
Exchange
Request Handling
Control
Request Handling
DI
DI
IDIP
ISIP
DI
DI
Cataloging
DI
DI
Dissemination
Ingest
Access
IDIP
DIP
SIP
Archiving
AIP
AIP
IDIP
ISIP
Processing
IDIP
SIP
SIP
Facility Ground Segment
Legend
Legend
AIP
SIP

Submission Information Package

SIP

Submission Information Package

Archival Information Package
ISIP

Internal Submission Information Package

ISIP

Internal Submission Information Package
IDIP

Internal Dissemination Information Package
DI

Descriptive Information
DI

DI

Descriptive Information
DIP

Dissemination Information Package
9
FGS Integration

• Following the logical model defined by the FEOMI
  project, the elements required to build the
  Facility Ground Segment for a specific mission
  are
• The so-called Core MMFI, i.e. a set of
  unconfigured multimission elements and services
  deployed in a suitable infrastructure.
• Several optional Mission Specific Elements
  (MSEs), accounting for the specificities of the
  missions sensors data, as seen before typically
  processing systems.
• The specific configuration of the MMFI Elements
  to allocate the specific services required by the
  mission, e.g. the ingestion and cataloguing of
  the specific datasets generated in the FGS.

10
Facility Ground Segment Decomposition
11
MMFI Architecture
12
MMFI Ingestion

• Controlled by the "Generic Front End" (GFE)
• Configurable workflow engine
• Registration of data producers and product types
• Ingestion of products with validation, metadata
  extraction, browse generation, etc.
• Generation of master catalogue (MMMC) update
  files
• Standard set of re-usable plug-ins relevant to
  ingestion
• Standard interface for the implementation of new
  plug-ins
• Access to data (including binary data) via Data
  Request Server (DRS)
• Metadata extraction and browse generation during
  ingestion based on the DRS

13
MMFI Data Library

• Based on the "Archive Management System" (AMS)
  and the "Local Inventory" (LI)
• The AMS manages the actual long-term archiving of
  the data products
• Abstraction layer toward underlying storage
  technology
• unique storage technology adopted by ESA for its
  EO missions, in order to achieve maximum
  harmonization and standardization
• Automation of operations
• On-line access via client-server services
• Internal storage organization transparent to
  clients
• Clients data access rights management
• Products subsetting to handle HBR data with EAST
  and DRB
• The LI is the local catalogue indexing all
  products archived in the long term archive at the
  centre
• Indexing and management of on-line archive
• UpLoad Server (ULS) for metadata synchronization
  with MMMC

14
MMFI Requests Handling

• Based on the Product Order Handling (POH)
  system
• Handles the on-demand production and
  dissemination requests received from the ESAs
  Multi-Mission Order Handling System (MMOHS)
• Organizes the required workflow based on the
  product type and output medium requested in the
  order
• Scheduling of data validation and retrieval from
  Data Library, processing, QC verification,
  dissemination to users
• Reports to MMOHS the Production Request statuses
• The POH is supported by a set of auxiliary
  components that interface other MMFI elements and
  provide specific functionality for workflow
  management

15
MMFI Dissemination

• Product Formatting and Delivery (PFD)
• Dissemination workflow management handling
  different dissemination channels in a
  configurable manner
• Optional reformatting and compression (different
  methods available, including wavelet/JPEG2000)
• Formatters with specific or generic plug-ins (DRB
  available for enhanced flexibility)
• Concurrent dissemination orders with priorities
  handling
• Media generation, on tape drives or CD/DVD. Small
  tape libraries for tapes and RImage CD/DVD
  Producer
• Generation of unique medium id with barcode
  printout, back inlay and delivery note for the
  end user
• PFD Network Server addition to perform electronic
  delivery with dynamic generation of random
  account and notification to user by e-mail.
  Notification back to POH?MMOHS of URL for user.

16
MMFI Dissemination (2)

• Product Distributor (PD)
• Based on PSM (Processing System Management)
  workflow engine
• systematic dissemination management (subscription
  and standing requests handling)
• systematic product dissemination (by timers and
  triggers)
• dissemination via PFD
• dissemination via DDS and other satellite
  multicast systems
• direct dissemination to ftp accounts and file
  locations
• product circulation to ftp accounts and file
  locations
• product circulation management with state based
  circulation control
• dissemination and circulation reporting

17
MMFI Processing

• PSM (Processing System Management)
• Main function is the abstraction of the
  processing systems to the higher level elements
  of the MMFI
• Allows integrating mission and sensor specific
  processing facilities with minimal effort and
  offering a choice of protocols for the definition
  of the MMFI - processing facility interface
• Also used in other elements thanks to the
  workflow model and interface with LI, able to
  perform complex scheduling of other elements
• PSM-CAR (Check And Release)
• PSM-PR (Product Retrieval)
• PSM-PD (Product Distributor)
• Powerful subscription mechanism to LI, based on
  OQL, to be notified of data appearance in the
  archive (systematic processing, reprocessing,
  circulation, etc.)

18
Systematic data-driven reprocessing
19
MMFI Use Cases - Circulation
20
MMFI Advanced Features

• The MMFI implements various features covering
  preservation and value adding concepts
• Support to operational scenarii for preservation
  strategies, e.g. periodic migration of digital
  products to new information technology
• Encapsulation by self-describing items as defined
  by OAIS model information packages. The
  maintenance of metadata is performed and means
  for data consistency are supplied
• Support for automated production by means of
  sophisticated data access and processing
  management
• Modular design, open architecture and streamlined
  interfaces that permit an easier substitution of
  one or more of its elements, if the need will
  arise in the future, to ensure the long-term
  preservation of its data holdings and of its
  services
• Special attention was put on the architecture for
  a well balanced assignment of functionality-to-com
  ponents.

21
MMFI System Design Features

• Archiving technology migration, by shielding the
  physical data-sets from the applications, using
  several software layers, that can all be used to
  handle lower level technology changes
• Hierarchical Storage Management (HSM)
  incorporated in the AMS. A potential change of
  the HSM technology would be fully resolved in the
  AMS.
• Limited number of components directly interfacing
  the AMS, thus also a change of the AMS or AMS
  interfaces could be handled with minimal archive
  impact.
• Technological evolution and scalability through
  the modularity, achieved by the architecture
  largely built from autonomous, networked
  components that can be combined by configuration
• Simplified exchange of some components by other
  implementations
• Handling of increased load by instantiating more
  components in optimised configurations (e.g.
  increase the number of processing nodes)
• Processor integration, to cope with changes to
  the Data Processors used for adding value to the
  EO data
• The PSM framework supports the integration of
  processor by natively supporting a variety of
  processor interfaces and by allowing integrating
  processor adapters
• The flexibility of this approach makes it
  possible to substitute processors and processor
  interfaces without undue effort and without
  affecting other parts of the participating
  workflows
• Product data model migration, to cope with the
  evolution of processing algorithms that requires
  changes in the data models of the products to be
  archived
• Configurable product object models within the
  Data Library that can be extended

22
MMFI Functional Features
23
MMFI Systematic Workflows