HYCOM Data Management

1

Integrated Management of Data for IOOS Modeling
and Analysis
Steve Hankin NOAA/PMEL Data Management and
Communications (DMAC) Modelers Caucus DMAC
Steering Team IOOS Modeling Workshop, Nov.
28-29 06
2
Why talk about DMAC and data management at a
modelers meeting?

• DMAC has been down a similar road
• DM is critical to the success of MA

3
Integrated Ocean Observing System
Observing Subsystem
Satellites
Aircraft
Fixed Platforms
Ships
Drifters Floats
AUVs
4
The DMAC Plan a community effort
- Arlie House workshop March 2002- DMAC SC
convened June 2002

• Brought together
• 6 Federal agencies
• 6 Universities
• 4 Regional/State agencies
• 3 Private sector orgs.
• Steering Committee guided2 Outreach Teams, 4
  Expert Teams
• Data Facilities Outreach
• User Outreach
• Data Discovery Metadata
• Data Transport
• Data Archive Access
• Applications Products

Public drafts circulated March 2003
5
The DMAC Plan a community effort

• Four levels of review
• Solicited expert review
• Public workshop
• Public email-based review
• Federal Registry comments
• Drafts broadly circulated
• ExCom, NORLC, IWG, USGST, Ocean.US Workshops,
  Regional Summits, Professional Society
  newsletters, email lists and websites
• 150 reviewers
• 6 Federal Agencies
• 22 Universities Institutes
• 13 Private Sector
• 13 Regional/State agencies
• 5 International

Published March 2005
6
The DMAC Plan a community effort

• An implementation plan (a roadmap to detailed
  specification)
• General - Overview (40 pp)
• Phased Implementation Plan (60 pp)
• Appendices (140 pages)

Includes Initial Concrete Guidance to Data
Providers
7
Competingcharacteristics rule out the most
straightforward solutions
8
No simple data standard can be designed that all
ocean data providers will utilize
9
No management structure exists to enforce
adoption of a complex, data management solution
10
Data cannot be centrally managed at a single
location
11
The DMAC Plan is a framework for interoperability
among independent, heterogeneous systems, large
and small

• Independence Organizations will manage their own
  data
• Standards enabling innovation!
• Web services the common language to be spoken
  by pre-existing systems

12
The greatest challenges are in the area of
community behavior, not technology
Corollary Even with(frustratingly) scarce
resources,meaningful progress is possible
The greatest challenges are in the area of
community behavior, not technology
13
The importance of standards

• What do we mean by a standard?
• an agreement (a document)
• a community of shared practice (using it)
• Standards enable innovation
• e.g. the Web

14
How DMAC selects standards

• Adapt, adoptand only as a last resortdevelop

Strong preference for open standards
15
DMAC Modelers Caucus

• Convened by the DMAC Steering Team
• Has identified data management needs thatlie on
  the boundary between MA and DMAC
• Some highlights follow

Members
16
Modelers CaucusDMAC issues for MAST

• CF-netCDF -- Climate and Forecast conventions
  CF is vital to model interoperability. Need
  extensions for
• Unstructured (finite element) grids and nested
  grids (critical to coastal modeling)
• GIS interoperability
• Nested grids

17
Community Standards for Unstructured
GridsUCAR/Unidata, Boulder CO -- gtgt October
16-18 ltltNOAA-IOOS funded

• Standards for unstructured coastal model outputs
• grid concepts, file contents, formats
• ?Standardize network access via OPeNDAP
• A community process
• demonstration testbed will becreated
• technical discussions are open
• http//cf-pcmdi.llnl.gov/

18
Modelers Caucus DMAC issues for MAST

• Tracking of hazardous waste spills,
  searchrescue, etc. require standards and best
  practices for
• Downscaling Conditioning the outputs of large
  scale models as BCs for coastal models
• Splining Models over neighboring coastal domains
  conditioned as input for a single problem domain

19
Modelers Caucus DMAC issues for MAST

• Standards for model metadata
• Model configuration.
• Model run parameters incl. BCs and ICs
• Certification (by whom? for what purposes?)
• Validation level, studies and data sources.
• QA/QC levels of outputs
• Forecast accuracy estimates
• Relevant projects ESMF, Curator,

20
Modelers Caucus DMAC issues for MAST

• OSSEs - Observing System Simulation
  ExperimentsModeling to improve Observing
  System design DMAC standards and services can
  facilitate OSSEs

21
Modelers Caucus DMAC issues for MAST

• Model-data and model-model comparisonModern
  data management can greatly accelerate modeling
  science by making a better laboratory
  environment.
• Much easier
• model-data validation
• modeling collaboration

22
Take-away messages

• Dont neglect data management.Improvements in
  data management will
• make models more usable by others
• strengthen the modeling community
• speed up model development
• Agree on standards and use them. Standards are
  empowering.
• Build your community.Even with scarce resources,
  meaningful progress can be achieved.

23
Discussion
24
Areas of data management vital to MA

• Model chaining (a.k.a. downscaling)
• Model intercomparison (made trivial)
• Modeling frameworks (components metadata)
• OSSEs (to help observing system design)
• Models better connected to decision support tools
  (to meet societal goals)