Integrated data management in the ESMF ESME

1
Integrated data management in the ESMF (ESME)

• Steve Hankin(NOAA/PMEL IOOS/DMAC)
• ESMF Team meeting
• July 2004

2
The growing importance of data integration to
modelers

• We no longer fund modeling.
• Today we fund climate prediction or coastal
  processes the science topics. Modeling is
  just a component.
• a program manager (anonymous)

3
Model outputs need to be made useful to many
classes of users
education
research community
modeling community
project
model run
4
Data and products need to be made more usable for
modelers
Real time and delayed-mode observations
assimilation
data products
modeling
boundary initial conditions
validation
comparison
5
An ESME must include a plan for data management.

• But, how ?
• Funds are limited

6
Partnership A community of data managers has
formed

• GO-ESSP Global Organization of Earth System
  Science Portals(http//esportal.gfdl.noaa.gov)
• Unidata
• ESG (NCAR, LLNL)
• OPeNDAP (a.k.a. DODS)
• COLA
• NOMADS (GFDL, PMEL, NCDC, NCEP)
• NASA/GCMD
• BADC, BODC
• WMO

7
Ocean data systems following similar approaches

• National Virtual Ocean Data System (NVODS)
• US Integrated Ocean Observing System
• GODAE (US and International)
• OCMIP, AOMIP,

8
Data portal components

• Data discovery
• Data access/transport
• On-line browse and comparison
• (Segue to analysis)

9
Data discovery Metadata search

• ? Mature standards do not exist today.
• A task for ESMF define and utilize metadata
  standards for modelers

10
Metadata for modelers

• Reviewed published (a standard)
• Structured (XML)
• Generated automatically in conjunction with
  setting up model runs
• Standardized parameter names (controlled
  vocabularies)
• Hierarchical
• components, grids, fields and attributes

11
Data discovery Metadata search

• Others are working hard on search
• Traditional metadata partners(e.g. GCMD)
• Semantic Web (Google on steroids)(3 years off?)

12
Data portal components

• Data discovery
• Data access/transport
• On-line browse and comparison
• (Segue to analysis)

13
Data transport

• OPeNDAP (a.k.a. DODS)
• Network data access
• Format-independence
• Subsetting
• Aggregation (GDS, Unidata)
• Compression
• Security Grid-enabled OPeNDAPg

14
OPeNDAP distributed access to data and semantic
metadata
15
CF (climate and forecast)

• CF 1.0 is now a standard
• use metadata e.g. units, coords.
• curvilinear, hybrid-Z, time-dependent
• great applicability beyond modeling, too
• Discussion question
• As the use of the CF standard widens how should
  the community support it?Not enough to endorse
  it. Need a partnership.

16
Data portal components

• Data discovery
• Data access/transport
• On-line browse and comparison
• (Segue to analysis)

17
Live Access Server
18
netCDF
19
LAS -- an Information Product Server
Ferret, CDAT or other

• Metadata (XML) contains the intelligence
• Back end applications do the real work
• OPeNDAP provides remote data access

20
(No Transcript)
21
Informationaccess
Uniform data access
22
A home page
Informationaccess
Uniform data access
Live Access Server
23
The UI talks to LAS through an XML web service
24
For example
25
On-line comparison
26
(No Transcript)
27
ExampleAverage over a lat-long box
28
Data portal components

• Data discovery
• Data access/transport
• On-line browse and comparison
• (Segue to analysis)

29
(No Transcript)
30
discover ? browse ? access
Metadata Standards
NASA Global Change Master Directory (GCMD)
31
Collaborating groups of modelers
LAS sisters share metadata to form a unified
(virtual) site.OPeNDAP allows LAS to difference
distributed fields.
32
v6.2 customizableinterfaces
33
Web Service access to products

• XML request protocol implemented and documented
• XML package out implemented and documented
• XML query protocol under development
• formal SOAP interface under development

34
A wealth of data products are available through
the National Virtual Ocean Data System (NVODS)
35
Example Zebiac model outputlive from IRI/LDEO
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Access to observations (WODB -- 9 million ocean
profiles)
40
(No Transcript)
41
(No Transcript)
42
configurable constraints
43
(No Transcript)
44
Batch access to products

• Query available data sets
• Query variables in data set model_1
• Query space-time domain
• Request a subset of data as a file (asc for
  ASCII format)

gtlasls http//cpu/LAS
gtlasls http//cpu/LAS  model_1
gtlasls http//cpu/LAS  model_1  sst
gtlasget  -x 2060  -y 2060 -t 11-Dec-2000  -f
asc        http//cpu/LAS  model_1 sst
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
IOOSUS Integrated Ocean Observing System

• Detect and Predict Change
• Mitigate natural hazards
• Improve safety and efficiency of marine ops
• Ensure national security
• Reduce public health risks
• Protect and restore marine ecosystems
• Sustain marine resources

49
IOOS Data Management andCommunications Subsystem
Ships
Hand Measurements
Satellites
Floats
Primary DataAssembly QC
Moorings
Metadata, Data Discoveryand Data
TransportStandards and Protocols
50
IOOS Data Management andCommunications Subsystem
Ships
Hand Measurements
Satellites
Floats
Primary DataAssembly QC
Moorings
RegionalData ManagementSystems
Products
Users
InternationalData ManagementSystems
Maps
Forecasts
Terrestrial and AtmosphericData
ManagementSystems
Metadata, Data Discoveryand Data
TransportStandards and Protocols
On-line Browse
Archive Centers
Modeling
51
Recommendations for ESMF

• Define ESMF metadata standard and use it
• Consider a GCMD modelers portal, too
• Endorse partnership with GO-ESSP and the emerging
  tools
• Data available through OPeNDAP(g)
• Live Access Servers for on-line collaborations

52
Questions?
NVODS LAShttp//www.ferret.noaa.gov/nvods GCMD
DODS Portalhttp//gcmd.gsfc.nasa.gov/Data/port
als/dods