State-of-the-art tools and practices for marine data and information management

1
State-of-the-art tools and practices for marine
data and information management a forward look

• Lesley Rickards
• British Oceanographic Data Centre (BODC)
• Chair, International Oceanographic Data and
  Information Exchange (IODE) Committee

2

• What are we aiming for?
• Simple access to all types of marine data (and
  information) on an appropriate time scale
• A virtual or distributed data centre
• Data available on CD-ROM/DVD
• End to end data management
• Where are we now?
• A collection of separate centres with a wide
  variety of remits, skills and data
• Made up of NODCs, RNODCs and WDCs
• Sometimes work together in groups for individual
  projects

3
(No Transcript)
4
Data Management and Communication System for the
coastal module of GOOS
5

• Data management practices
• Best rather than state of the art
• Compliance with IOC Data Exchange Policy
• Proper collaborative efforts, building on
  existing standards and practices not
  reinventing the wheel each time a new project
  comes along
• Cooperation partnership collaboration

6

• Timely, efficient and open access to the best
• possible data, metadata associated products
• Metadata Standards
• Discovery
• Accompanying data (position, date/time, etc)
• Data Documentation (qc history)
• Quality Control
• Automatic tests
• Scientific, delayed-mode
• Data dissemination
• CD-ROM/DVD
• On-line access
• Long term stewardship of data

7

• Data management tools - METADATA
• Metadata is all the descriptive information
  necessary to allow users to find (discover),
  access, manipulate, process (request) and extract
  (recover) data, information and products
• Standards
• ISO19115
• Dublin Core
• MEDI, GCMD, EDMED, FGDC, ANZLIC, (EDIOS), (CSR),
  etc, etc.
• Mappings between standards available
  (cross-walks)
• Compliance with ISO19115
• Various controlled vocabularies
• Links with XML (schemas/DTDs)
• MEDI Authoring Tool
• All have search tools
• geospatial location, temporal information,
  keywords, controlled vocabulary items, free text

8

• eXtensible Markup Language (XML)
• XML is being widely used as a basis for both
  dynamic web page development and more generally
  as a data exchange mechanism
• Data exchange aspects of XML include the ability
  to define flexible data structures that utilise
  the terminology of the subject area
• Data to be exchanged is packaged in a form more
  intuitive to the user.
• Extensive availability of free software for
  manipulation and transformation of the XML data
  stream
• Allows developers to easily develop, populate,
  exchange and transform data streams.

9

• MarineXML
• EU MarineXML
• demonstrate how XML technology can be used to
  develop a framework that improves the
  interoperability of data for the marine community
  and specifically in support of marine observing
  systems. The project will develop a prototype of
  an XML-based Marine Mark-up Language (MML).
• ICES-IOC SGXML
• utilize or establish international standards
  to promote the seamless exchange of data from
  distributed data sources, by using a single
  parameter dictionary, well-defined and explicitly
  tagged metadata, and a common XML data structure,
  packaging all content and providing to the client
  datasets and software tools that are platform
  independent or web enabled

10

• Quality Control
• (Real-time) automatic tests (but do not rely on
  them)
• Quality flags
• Data visualisation tools
• to include comparison with other data collected
  in the same place or nearby, climatology
  different ways of looking at the same data
• Ocean Data View
• Ncbrowse (for netCDF files)
• EPIC (management, display and analysis of
  oceanographic and meteorological data)
• Sea Level data
• POL TASK2000 package on-line tidal analysis
• University of Hawaii JASL software
• ESEAS working towards new package
• Document QC (e.g. audit trail, data history)
• Use existing standards/guidelines where available
  
• ICES data type guidelines, WOCE standards

11

• Ocean Data View (ODV)
• Interactive exploration and graphical display of
  oceanographic and other geo-referenced profile,
  sequence or gridded data
• Runs on Windows (9x/NT/2000/XP), Linux, UNIX, and
  Mac OS X
• Data collection and configuration files are
  platform independent
• Interactive browse through large sets of station
  data

12

• Ocean Data View (ODV)
• High-quality station-maps, general
  property-property plots of one or more stations,
  scatter plots of selected stations, property
  sections along arbitrary cruise tracks and
  property distributions on general iso-surfaces
• Display of original scalar and vector data by
  coloured dots, numerical data values or arrows
• Fast gridding algorithms allow colour shading and
  contouring of gridded fields along sections and
  on iso-surfaces
• Derived quantities calculated dynamically,
  displayed and analysed

13
(No Transcript)
14

• Development of distributed systems
• OPeNDAP (DODS)
• Data servers for making local data accessible at
  remote locations
• Free software for download
• Live Active Server (LAS)
• Best for large, gridded environmental data sets
• Dynamically generated graphics
• Compare variables from distributed locations
  (using DODS)
• Thematic Real-time Environmental Data Distributed
• Services (THREDDS)
• Access to large collection of real-time and
  archived data sets from a variety of data sources
• Analysis and display software

15

• Other examples of distributed systems
• US GLOBEC (US JGOFS)
• NOAA Coastal Directory
• Russian ESIMO
• NERC Data Grid
• IFREMER/SISMER Data Portal
• SeaSearch Common Data Index
• Distributed Generic Information Retrieval (DiGIR)
  protocol for retrieving structured information
  from multiple heterogeneous databases
• etc.

16
SISMER Web Portal
NOW
SISMER WEB interface
WEB interface 1
WEB interface i
WEB interface 2
SISMER Server Database Datasets catalogues 
Thematic database i
Thematicdatabase 1
Thematicdatabase 2
SISMERWEB portal
FUTURE SISMER Web Portal
Automatic querying of existing systems
SISMER Server Database Data sets catalogues 
XML / ISO 19115 integrator 1
XML / ISO 19115 integrator 2
XML / ISO 19115 integrator i
Thematicdatabase 2
Thematicdatabase 1
Thematic database i
17
ARGO STATUS (Jan 27, 2004) 1037 Active Floats Target 3000 floats by 2006  
18

• Argo Data Management
• Data transmitted in real-time by satellite
• Transferred to data centres
• Messages decoded
• Automatic real-time quality control tests
• Data passed to global centres (GDACs) for
  dissemination
• Delayed-mode quality control and calibration
• Replacement version sent to GDACs
• Regional centres for further quality control and
  products

19

• Location of Argo float profiles
• 01 27 Jan2004
• No. of profiles
• 2139
• Data available for download
• Profiles
• Trajectory
• Metadata
• Technical info
• From
• US GODAE
• CORIOLIS

20
Ocean Biogeographic Information System (OBIS)

• Web-based provider
• Global geo-referenced information
• Accurately identified marine species
• Expert species level and habitat level databases
• Variety of spatial query tools for visualizing
  relationships among species and their
  environment.
• Strives to assess and integrate biological,
  physical, and chemical oceanographic data from
  multiple sources.
• Users, including researchers, students, and
  environmental managers, gain a dynamic view of
  the multi-dimensional oceanic world
• Part of the Census of Marine Life (CoML)

21

• Gulf of Maine Biogeographic Information System
• (GMBIS)
• Regional implementation for CoML
• Partner with OBIS
• Integrated into the Gulf of Maine Ocean Observing
  System (GoMOOS)
• Designed to assimilate and integrate marine
  ecosystem and fisheries data, as well as
  natural-history information
• Included is an advanced oceanographic geographic
  information system (GIS)

22

• Gulf of Maine Biogeographic Information System
• (GMBIS)
• Access to biological, physical, chemical and
  geological data and information
• Enhance understanding of biological patterns and
  their changes through time
• Aggregation server providing access, rapid
  visualization and data download capabilities
• Server will rely on a combination of archived
  (local access) data as well as dynamic access to
  remote data providers
• Visualisation and other interactive software will
  be designed to help the user evaluate what data
  are available, combine data layers and download
  data

23
Gulf of Maine Biogeographic Information System
(GMBIS)

• Access to
• historical data
• taxonomic collections
• geological base maps
• modern remote sensing data (buoys, CODAR,
  satellite)
• modelling products
• broad-scale survey data (e.g. living marine
  resources)
• other monitoring programs (e.g. COASTWATCH,
  Continuous Plankton Recorder)

24

• GMBIS uses Environmental Analysis System (EASy)
• Storage, dissemination, analysis, integration,
  and dynamic display of spatially referenced
  series of oceanographic data
• PC-based
• Aids interfacing of multivariate oceanographic
  data
• Both data and model outputs can be imaged in time
  through diverse kinds of displays
• Including vector, contour, and false-colour
  imagery
• Vertical structures can be depicted along line
  transects or point-sampling stations
• Time series can also be visualised
• Patterns in the spatial distribution or organisms
  to be visualized and compared to other spatial
  distributions
• Including both biological and other oceanographic
  variables
• Even when characterised by different scales of
  sampling and different degrees of resolution

25

• CONCLUSIONS
• Use of modern IT techniques in a transparent
  manner to improve service to users (internet,
  web, distributed systems, XML, etc.)
• Setting the standards (metadata, data quality
  control, data stewardship)
• Working collaboratively whilst responding to
  national remit
• Increasing data diversity (many different
  parameters being measured)
• Developing systems to deliver (near) real-time
  data