Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered in GALEON results or discussion of access control

1
Recent developments with the THREDDS Data Server
(TDS) and related Tools covering TDS, NCML, WCS,
forecast aggregation and not including stuff
covered in GALEON results or discussion of access
control

• John Caron
• UCAR/Unidata

2
A few Good Things to know about the THREDDS Data
Server

• John Caron
• UCAR/Unidata

3
THREDDS Data Server

• 100 Java
• Runs within Tomcat
• Data Server, not a Data Portal
• no pictures
• html interface
• Production status
• Early adopters - Cutting but not bleeding edge
• Starting to look at performance issues

4
TDS 5 things to know

• Reads CDM files
• Serves OPeNDAP, WCS, NetCDF
• Configured with THREDDS Catalogs
• Integrated with NcML
• Augment metadata
• Virtual Datasets / Aggregation
• Connects to Digital Libraries through OAI

5
THREDDS Data Server
OAI Harvester
HTTP Tomcat Server
OAI Provider
DL Records
Catalog.xml
THREDDS Server
Application
NetCDF-Java (CDM) library

• OPeNDAP

• HTTPServer

• WCS

Datasets
otherhost.gov
OPeNDAP Server
hostname.edu
6
1. Reads CDM files

• NetCDF-3
• HDF5 (most)
• OPeNDAP client/server protocol
• GRIB version 1 and version 2 (ongoing work)
• BUFR (limited - ongoing work)
• GINI (GOES Ingest and NOAAPORT Interface ) image
  format
• DMSP (Defense Meteorological Satellite Program)
• NEXRAD Radar level 2 and level 3.
• ATD's DORADE radar file format
• ADDE/McIDAS client/server protocol (limited)

7
Application
Scientific Datatypes
Datatype Adapter
CDM Architecture
Coordinate Systems
CoordSystem Builder
Data Access
ADDE
I/O service provider
OPeNDAP
NetCDF-3
BUFR
GRIB
NetCDF-4
HDF5
GINI
NEXRAD
DSMP

8
2. Serves OPeNDAP, WCS, NetCDF

• OPeNDAP
• remote data access to CDM datasets
• WCS gridded data only
• CDM must understand geo coord system
• Data transferred as GeoTIFF, netCDF-CF
• Serve raw data no re-projections
• NetCDF subsetting service (experimental )
• request lat/lon/time bounding box
• return netCDF-CF files
• HTTP bulk file transfer

9
3. Client sees THREDDS Catalogs

• Plain old XML documents - no search or other
  capabilities
• Logical directory tree of on-line resources
• Place to add metadata

10
3. TDS Configured with Catalogs

• ltcataloggt
• ltservice namedods" serviceType"OpenDAP"
  base"/thredds/dodsC/ /gt
• ltdatasetScan nameNEXRAD serviceNamedods
• pathradars" location/data/radar/ /gt
• lt/cataloggt

11
4. Integrated with NcML

• ltdatasetScan namemodels serviceNamedods
• pathmodels/ocean" location/data/models/
  /gt
• ltnetcdf gt
• ltattribute nameConventions
  valueCOARDS/gt
• ltvariable name Temperature gt
• ltattribute nameunits valueK/gt
• lt/variablegt
• lt/netcdfgt
• lt/datasetScangt

12
4. Integrated with NcML

• ltdataset name"WEST-CONUS_4km Aggregation"
  urlPath"satellite/3.9/WEST-CONUS_4km"gt
• ltnetcdfgt
• ltaggregation dimName"time"
  type"joinExisting"gt
• ltscan locationC/data/goes/"
  suffix".gini"/gt
• lt/aggregationgt
• lt/netcdfgt
• lt/datasetgt

13
5. Connects to Digital Library through OAI

• Enhanced Catalogs enough metadata to create
  Digital Library Records
• Crawl catalog and create records
• Metadata is in the public catalogs so anyone
  can harvest
• Open Archives Initiative (OAI) harvesting
  protocol (DLESE)
• DIF (GCMD) and ADN (DLESE) records
• Seems like ISO 19139 worth doing also

14
Whats Next?

• DataType Aggregation
• Automatic extraction of metadata from datasets
• LAS, OPeNDAP, CDP, NOMADS, NGDC
• Spring(?)

15
DataType Aggregation

• Work at the CDM DataType level, know (some) data
  semantics
• Forecast Model Collections
• Combine multiple model forecasts into single
  dataset with two time dimensions
• With NOAA/IOOS (Steve Hankin)
• Point/Station/Trajectory/Profile Data
• Allow space/time queries, return nested sequences
• Start from / standardize Dapper conventions

16
Forecast Model Collections
17
TDS/CDM training workshop

• Advanced uses of netCDF and TDS
• Oct 30 Nov 2, 2006
• NetCDF C library (1 day)
• NetCDF Java library (1 day)
• TDS (2 days)
• Boulder, Colorado
• Free, limited space
• Video will be available

18
TDS Access Control

• ltdataset nameGOES West 4km Visible"
  urlPath"satellite/WEST-CONUS_4km
• resourceControlgroupName
• gt
• Dont know how to trigger authorization challenge
  from servlet

19
TDS resource control

• Track users resource consumption
• Session cookies
• Track by IP
• Customer categories
• Preferred top priority
• Registered we know who you are
• Normal IP only, use spare cycles, with limits
• Abusers ignore, penalty box, etc.

20
Data Access Standards

• Data Subsetting
• What queries are allowed?
• What queries can be answered efficiently?
• Data Encoding
• What does the result of the query look like?
• On-the-wire
• Through the API (language specific)

21
Standardization

• Start by designing an abstract model.
• OGC/ISO, GML.
• Comprehensive
• Danger Premature standardization, wrong
  tradeoffs
• Start from current usage
• CDM netCDF, HDF5, OPeNDAP CF-1.0
• 20 covers 80 of whats needed
• Danger ad-hoc, fragmentation