Implementation and Plans for TIGGE Phase I at NCAR and ECMWF

1
Implementation and Plans for TIGGE Phase I at
NCAR and ECMWF

• Doug Schuster
• Computational Information Systems Laboratory
  (CISL) at The National Center for Atmospheric
  Research (NCAR)

2
Outline

• Archive Center Data Collection
• Data Discovery, Access, And Distribution
• Analysis Tools
• Future User Services
• Summary

3
Archive Center Data Collection

• Participating Archive Centers
• National Center for Atmospheric Research (NCAR)
• European Center for Medium Range Weather
  Forecasts (ECMWF)
• China Meteorological Administration (CMA)
• Data Collection Mechanism
• Unidatas Internet Data Distribution/Local Data
  Manager system (IDD/LDM)
• History of providing similar functionality in
  delivering National Center for Environmental
  Prediction (NCEP) model data to the university
  community.

4
Archive Center Data Collection

• Current Data Providers
• ECMWF (10/1/2006)
• United Kingdom (10/1/2006)
• Japan (10/1/2006)
• NCEP (US) (11/1/2006)
• Future Data Providers
• Australia
• China
• Canada
• Brazil
• Korea
• France

5
Archive Center Data Collection

• TIGGE Parameters
• Available on 4 level types
• Single Level (includes surface)
• Pressure Level (1000, 925, 850, 700, 500, 300,
  250, 200 hPa, with geopotential height only at 50
  hPa)
• Isentropic Level (320K)
• Potential Vorticity Level (2 PVU)

6
Archive Center Data Collection

• TIGGE Archive at NCAR
• Model output stored as forecast files for each
  data provider
• Each file contains all parameters and ensemble
  members for a level type and forecast time
• 2 3 week period of most recent forecast files
  online for direct user download.
• Complete archive maintained on the Mass Store
  System as part of the CISL Research Data Archive
  (RDA).
• TIGGE Archive at ECMWF
• Data archived through ECMWFs MARS system.
• Stores individual GRIB messages.

7
Archive Center Data Collection

• Summary of Data Providers

8
Archive Center Data Collection

• Summary of Data Provider Forecasts

9
Archive Center Data Collection

• Pressure Level Parameters

10
Archive Center Data Collection

• Isentropic Level Parameters

Potential Vorticity Level Parameters
11
Archive Center Data Collection

• Single Level Parameters

12
Archive Center Data Collection

• Single Level Parameters cont.

13
Archive Center Data Collection

• Single Level Parameters cont.

14
Archive Center Data Collection

• Single Level Parameters cont.

15
Archive Center Data Collection

• Single Level Parameters cont.

16
Data Discovery, Access, and Distribution

• NCARs TIGGE Web Portal (http//tigge.ucar.edu)
• Users can search, discover, and download forecast
  files
• Select data by initialization date/time, data
  provider, parameter level, and forecast time.
• Each file contains all parameters and ensemble
  members for a given level type
• Single Level Parameters (Surface)
• Pressure Level Parameters
• Potential Vorticity Level Parameters
• Potential Temperature Level Parameters.
• Contains most recent 2-3 weeks of model output.
• 48 hour data access delay from model init time.

17
Data Discovery Access and Distribution
18
Data Discovery Access and Distribution
19
Data Discovery Access and Distribution

• NCAR Research Data Archive
• http//dss.ucar.edu/datasets/ds330.0/ -2006
• http//dss.ucar.edu/datasets/ds330.1/ -2007
• The RDA dataset enables
• Access to the complete TIGGE archive.
• Easy use of the TIGGE archive on NCAR CISL
  computers data coming directly from the Mass
  Store System (MSS)
• Support staff to handle requests for offline data
• Forecast file structure identical to online files

20
Data Discovery Access and Distribution
21
Data Discovery Access and Distribution
22
Data Discovery Access and Distribution
23
Data Discovery Access and Distribution
24
Analysis Tools

• User Analysis and Basic Data Manipulation Tools
• WMO GRIB2 is relatively new, so tools are
  immature
• Forecasts with ensemble members add another
  dimension
• Improvements underway in
• NCAR Command Language (NCL)/Python
• Gempak, GRIB-Java (Unidata)
• NCEP tools (wgrib2, etc)
• ECMWF GRIB API
• http//tigge.ucar.edu/data-tools.htm

25
Future User Services

• TIGGE Portal 2007 Improvements
• Develop streaming download for multiple files
• Upgrade to handle subset data requests through a
  simple interface (e.g. parameter selection)
• Provide user selected grid interpolation across
  multiple models
• Add spatial sub-setting functionality
• Include subscription services for recurring
  requests
• Provide web services for automated requests
• Common interface for NCAR, ECMWF, and CMA

26
Summary

• ECMWF and NCAR now archiving data from 4
  providers (ECMWF, NCEP, UKMO, JMA)
• 6 more data providers plan to come online over
  2007 (Australia, Brazil, Canada, China, Korea,
  France)
• The TIGGE archive and access system designed to
  accommodate irregularity between providers.
• Desired 2007 Improvements include
• Streaming download for multiple forecast files.
• User specified sub-set/grid interpolation
  requests across multiple models.
• A common web service interface for CMA, ECMWF,
  and NCAR for automated requests.

27
TIGGE Portal / Web Services Architecture

• Java 5 based implementation.
• Developed with the Spring application framework.
• Web Portal Interface
• Point and click interface allows the selection
  and downloading of forecast dataset.
• Web Services
• REST based Web Services to allow clients to
  discover and download data.
• Provides an interface for automated clients to
  download data in large volumes and at regular
  intervals.

28
REST Web Services

• REST Representational State Transfer.
• REST is not a standard for web services, it is a
  style of web service.
• REST is built on existing standards
  (HTTP/URL/XML).
• Requests are sent as valid HTTP requests using
  the appropriate verbs (GET, POST, PUT, DELETE).
• Responses are returned service specific XML
  documents.
• Since REST is not a standard it avoids some of
  the pitfalls and problems associated with SOAP
  based web service interoperability.
• In this particular application it allows for
  returning complex data types that are not
  inherently bound to implementation
  toolkit/language.

29
TIGGE Data Selection/Download Architecture
Tomcat Engine
MySql
TIGGE Forecast Metadata Database
Forecast Metadate Database Access
Browser Client
TIGGE Web Portal Interface
HTTP Request/Response
TIGGE Forecast Catalog Service
Web Service Client
TIGGE Web Service Interface
HTTP/XML Rest Request/Response
GRIB2 Forecast Data Files
30
TIGGE Data Subsetting Architecture
Tomcat Engine
MySql
TIGGE Subset Request Database
Browser Client
TIGGE Web Portal Interface
Forecast Metadate Database Access
HTTP Request/Response
TIGGE Forecast Catalog Service
Web Service Client
TIGGE Web Service Interface
HTTP/XML Rest Request/Response
Perl Server
Embedded HTTP Client
Data Extraction Engine
31
Archive Center Data Collection

• Current TIGGE Data Ingest Status
• Data is coming from three separate IDD/LDM
  systems
• ECMWF (Data Provider ECMWF, UKMO, JMA)
• Overwhelmingly the largest
• CONDUIT (NCEP), maintained by Unidata
• CPTEC (Brazil)

ECMWF (UKMO)
JMA
CPTEC
NCEP
Unidata Server
NCAR
32
Archive Center Data Collection
Hourly Volume Receipt from IDD EXP Stream
33
Archive Center Data Collection
Hourly Volume Receipt from IDD CONDUIT Stream
34
Archive Center Data Collection

• Hourly Cumulative Volume Data Summary

35
Archive Center Data Collection

• Highlights
• Succeeded to reach target data rate (10 GB/hr)
  Oct. 2005
• Build up supporting systems and software around
  IDD/LDM
• Develop archiving procedures and implement file
  management for the TIGGE portal.
• Begin realistic daily test data flow April 2006
• Initiate operational data collection October 1,
  2006

36
Future User Services

• Sub-setting and Grid Interpolation
• Users will need more than file downloads at model
  center native resolution
• Specify parameter, temporal, and spatial
  sub-setting across all models
• Requires verified software from each data
  provider
• Must build and run effectively on local hardware

37
TIGGE - Who, What, Why?

• WMO THORPEX Interactive Grand Global Ensemble
  (TIGGE)
• Foster multi-model ensemble studies.
• Improve accuracy of 1 to 14 day high-impact
  weather forecasts.
• Up to 10 operational centers contributing
  ensemble data.
• Two Phases
• 1 Three central Archive Centers
• China Meteorological Administration (CMA)
• European Center for Medium-Range Weather
  Forecasts (ECMWF)
• National Center for Atmospheric Research (NCAR)
• 2 Widely Distributed Access (not discussed
  today)

38
Archive Center Data Collection

• Cooperative Support Team for Data Ingest and
  Highlights
• ECMWF Raoult, Fuentes
• Optimally tune ECWMF and NCAR systems to work
  together
• Develop protocols to ensure complete transfer
  (e.g. grid manifests)
• Unidata Yoksas
• Expert advise on IDD/LDM
• VETS Brown
• Installed, configured/re-configured, monitored
  IDD/LDM
• DSG Arnold
• System administration
• NETS Mitchell
• TCP packet analysis and advise

39
Current and Future Challenges

• Transition from Dataportal to Ultra-zone was not
  seamless
• Machine differences perturbed a number of
  adjustments
• Clock synchronization with ECMWF
• TCP Cache, and File IO settings
• LDM configuration
• Things are running well now.

40
TIGGE Status _at_ NCAR

• Data Received
• Data transport by Unidatas IDD/LDM application
• Tuned to a maximum of about 10 GB/Hour (optimum
  rate between ECMWF and NCAR)
• Transfer packets are individual GRIB2 messages
  (single 2D fields)
• Current receipt metrics, 172 GB/Day, 809K
  fields
• Data organization and storage
• Fields are combined into forecast files organized
  by
• Initialization time, data provider, level type,
  and forecast time step
• All ensemble members are included in each
  forecast file
• Most current three-week period kept online
  (4-6TB)
• Long-term times series archived on the NCAR Mass
  Storage System
• This file based approach contrasts with the
  ECMWF MARS approach

41
TIGGE Status _at_ NCAR

• Data Providers
• Current participation
• N200, Reduced Gaussian
• There are 71 standard grids, varying levels of
  compliance now
• Grids 26 surface or single level 45 pressure
  level
• Future contributors
• Australia, Brazil, Canada, China, Korea, and
  France
• Archive System is designed to handle varying
  forecasts per day, output resolution, and
  numbers of ensemble members

42
TIGGE Status _at_ NCAR

• Next Steps
• Design and implement user registration
• Simple online form, agree to research and
  education work only
• Default, 48 hour delay (from forecast init.
  time) before access
• Real-time access granted by International
  Program Office
• E.g. Special field project support
• Open user access
• Initially, only file download through a portal
  interface
• http//tigge.ucar.edu (to be activated early
  November)
• User Software, a challenge because GRIB2 is new
• Put examples and resource pointers into portal
  (end November)
• Improvements will be posted as they become
  available
• User defined subsets and regridding across
  multiple models
• Development depends on securing additional
  funding
• Bring new Data Providers online as they become
  ready