NOAA Science Advisory Boards Data Archiving and Access Requirements Working Group DAARWG May 2425, 2

1
Overview of NOAA Archive Policy
Tom Karl Larry Tyminski Co-Chairs NOAA Data
Management Committee
NOAA Science Advisory BoardsData Archiving and
Access Requirements Working Group (DAARWG)May
24-25, 2007Chicago, IL
2
Briefing Outline

• NOAA Archive Policy
• Overview NOAA Decision Processes
• What to archive and how long
• NOAA Archive Policies
• Data Centers Centers of Data
• Five NOAA Line Office Policies
• NOAA archive issues and what is expected in the
  future
• Data versions how many to keep?
• Exponential data volume increase
• CLASS data storage for all NOAA data?

3
NOAA Data to ArchiveThree Archive Considerations

• Legal Requirements
• Data required by law to be archived

• Practical Requirements
• Data/products which require long processing times
  to reproduce
• Non NOAA Observations that support NOAAs
  mission
• Policy Requirements
• Data for Regulatory purposes or used in
  Scientific Assessments

NOAA responsibility DAARWG
recommendations (Note Multiple versions of
data/ products exist in all three areas,
(especially 2) due to reanalyses, QC levels, etc.
What versions to keep are open for DAARWG
recommendations)
4
Which NOAA Data to Archive?Consideration 1
Legal Requirements
U.S. National Archives Records Administration
(NARA) Policy - preserving records that contain
adequate and proper documentation of the
organization, functions, policies, decisions,
procedures, and essential transactions of the
agency

• NOAA Data Archive Legal Requirement Examples
• Weather/Climate
• Original In-situ observations, weather forecasts,
  satellite, radar
• Oceans
• Original Sub surface observations, Water levels
  and tides
• Ecosystem observations (Coral reefs)
• Solid Earth
• Seismic, Volcanic, Magnetic, Bathymetric,
  Glacial, Nautical Charts, Shoreline
• Space
• Solar, Terrestrial Physics

NARA policy requires that a Records Disposition
Schedule list the types of records maintained by
each department of an organization, and specify a
period of time records are to be retained for
operational, legal, fiscal, historical, or other
purposes.
5
Which NOAA Data to Archive?Consideration 2
Practical Requirements
NOAA Archive Practical Requirement Examples

• Certain data products require large processing
  times to reproduce
• Model output Products produced from original
  observations
• Examples Weather and climate models, Climate
  Model based Reanalyses
• Satellite Products produced from geo-located
  observations
• Examples Vegetation Index, blended sea winds,
  blended sea surface temperatures, nighttime
  lights, ocean color, etc.
• Radar Level-III NEXRAD products
• Examples Base reflectivity, Echo tops, One-hour
  precipitation, etc.
• Non-NOAA Observations that support NOAAs mission
• Satellite NASA, international
• Other International data via WMO, World Data
  Centers, etc.

Multiple versions of data and products exist in
this area
6
Which NOAA Data to Archive?Consideration 3
Policy Requirements
Data for Regulatory Purposes and Reproduce
Scientific Assessments

• Examples of Policy Made by Government Agencies
  using NOAA Data
• Fisheries data used in policies on fish size and
  bag limits
• US Army Corp Engineers use of revised hurricane
  statistics for rebuilding New Orleans levees
• Climate Normals used by Utilities in setting
  power rates
• Data used in Scientific Assessment must be
  archived
• U.S.
• Climate Change Science Program Assessments
• State of the Climate Reports
• International
• International Panel on Climate Change (IPCC)

CCSP 1.1 Assessment Released March 31, 2006
7
NOAA Archive Policies
8
NOAA Archive StructureData Centers Centers of
Data

• NOAA Data Centers Archive 95 of NOAAs data
• NCDC, NODC, NGDC and CLASS (data storage)
• Similar standards used for data/metadata
• NARA policies are implemented
• NOAA Centers of Data Archive 5 of NOAAs
  data
• Many (dozens) NOAA offices are official Centers
  of Data (e.g., National Snow and Ice Data Center)
• Almost all NOAA offices (100s) contain smaller
  data sets critical to their mission
• Exception high volume archives of climate and
  weather models
• Some standards for data / metadata
• NARA policies may not be implemented (e.g., no
  back-ups, non standard formats and media)

9
NOAA Line Office Archive PoliciesNESDIS and NWS

• National Environmental Satellite Data
  Information Service
• Archive a core mission via 3 data centers NCDC,
  NODC, NGDC
• NESDIS data centers archive some data for all
  NOAA Line offices
• CLASS plans archival storage for all NOAA data
  starting with large array
• National Weather Service
• NCDC performs almost all NWS archive functions
  for data which may be needed at a later time for
  accident investigations and litigation
• Original transmitted observations taken by NWS,
  FAA, or DOD
• Analysis and Forecast products officially
  transmitted by NWS
• In the event of litigation, NWS offices maintain
  misc. records (e.g. non transmitted records,
  station duty manuals) Offices also save weather
  cases for training
• For large array data sets NCDC (stewardship)
  and/or CLASS (storage) provide archival services
  for radar observations (NEXRAD), some NCEP
  weather models, Satellites (GOES, POES)

10
NOAA Line Office Archive PoliciesNMFS, NOS, and
NOAA Research

• National Marine Fisheries Service
• Six NMFS Centers of Data manage small volume,
  high content data
• NESDIS (NODC) provides archive and access for
  some NMFS data
• Restricted data access due to privacy issues of
  data provider and regulatory nature of data
• National Ocean Service
• Two NOS Centers of Data manage geodetic and water
  level data
• Relies on many archived observations provided by
  other NOAA Line Offices
• Manages data via NOS Centers of Data and NESDIS
  data centers (NODC, NGDC)
• NOAA Research
• Maintains large volume modeling data bases at
  Centers of Data (NCEP,GFDL, ARL)
• NCDC archives of some climate and weather model
  outputs
• Trace Gas archives managed by DOEs World Data
  Center for Trace Gases

11
NOAA archive issues and what is expected in the
future
12
NOAA Archive Issues
Issue How many data set versions to keep?
Example 1

• Example Data Set Evolution NOAAs Microwave
  Sounder Unit

• History of data set adjustments based upon
  scientific discoveries
• Data set improvements occur when data are
  regularly used more thoroughly scrutinized

13
NOAA Archive Issues
Issue How many data set versions to keep?
Example 2

• Freedom of Information Act (FOIA) Requests
  Retention of NOAA information used in scientific
  research or policy regulations

• 2007 FOIA Request
• Data requested from 1990 climate research on
  urbanization effects
• Data requested by international researcher

14
NOAA Archive Issues
Exponential increase in NOAA data volume
projected from systems such as NPP, NPOESS, and
new model simulations
NOAAs Environmental Data Archive

• Broad Issues
• Annual CLASS data storage budget is in the low
  million range
• Will the decreases expected in data storage cost
  keep pace with exponential increase in CLASS data
  volumes?
• NOAA has some capability to provide data
  stewardship for existing data
• Will NOAA be able to provide data stewardship to
  produce new Climate Data Records from planned
  satellite missions?
• GEO IDE to provide standards and protocols for
  interoperability is unfunded
• Will data access and inter-disciplinary analysis
  be feasible without GEO IDE?

15
NOAA Archive Issues
NOAA has identified CLASS as the primary system
to meet NOAAs data storage requirements.

• CLASS Present
• Provide data storage for large array data
  systems (currently GOES/POES Satellites, NODC
  smaller data sets as a prototype)
• CLASS Future
• Provide storage for future large array data.
  Also, provide storage for NOAAs smaller data
  sets (e.g., biological physical). Specific
  language
• The current CLASS program supports the archiving
  of the specific and limited set of large array
  data sets identified above. Other major
  observing campaigns not listed above that have
  significant identified archiving requirements are
  directed to plan to use CLASS to meet their
  requirements and to provide additional resources
  to the CLASS program for archiving services.
  Alternately, they can use the CLASS enterprise
  approach to develop data management systems, but
  exemptions to this policy must be granted by the
  NOSC and the CIO Council.

Issue Is this NOAA policy reasonable for 1)
large array data sets and 2) smaller
multi-discipline data sets?
16
Questions