Implementation and Plans for TIGGE Phase I at NCAR and ECMWF - PowerPoint PPT Presentation

Loading...

PPT – Implementation and Plans for TIGGE Phase I at NCAR and ECMWF PowerPoint presentation | free to view - id: 20064e-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Implementation and Plans for TIGGE Phase I at NCAR and ECMWF

Description:

European Center for Medium Range Weather Forecasts (ECMWF) ... Long-term times series archived on the NCAR Mass Storage System ... – PowerPoint PPT presentation

Number of Views:32
Avg rating:3.0/5.0
Slides: 43
Provided by: ncar7
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Implementation and Plans for TIGGE Phase I at NCAR and ECMWF


1
Implementation and Plans for TIGGE Phase I at
NCAR and ECMWF
  • Doug Schuster
  • Computational Information Systems Laboratory
    (CISL) at The National Center for Atmospheric
    Research (NCAR)

2
Outline
  • Archive Center Data Collection
  • Data Discovery, Access, And Distribution
  • Analysis Tools
  • Future User Services
  • Summary

3
Archive Center Data Collection
  • Participating Archive Centers
  • National Center for Atmospheric Research (NCAR)
  • European Center for Medium Range Weather
    Forecasts (ECMWF)
  • China Meteorological Administration (CMA)
  • Data Collection Mechanism
  • Unidatas Internet Data Distribution/Local Data
    Manager system (IDD/LDM)
  • History of providing similar functionality in
    delivering National Center for Environmental
    Prediction (NCEP) model data to the university
    community.

4
Archive Center Data Collection
  • Current Data Providers
  • ECMWF (10/1/2006)
  • United Kingdom (10/1/2006)
  • Japan (10/1/2006)
  • NCEP (US) (11/1/2006)
  • Future Data Providers
  • Australia
  • China
  • Canada
  • Brazil
  • Korea
  • France

5
Archive Center Data Collection
  • TIGGE Parameters
  • Available on 4 level types
  • Single Level (includes surface)
  • Pressure Level (1000, 925, 850, 700, 500, 300,
    250, 200 hPa, with geopotential height only at 50
    hPa)
  • Isentropic Level (320K)
  • Potential Vorticity Level (2 PVU)

6
Archive Center Data Collection
  • TIGGE Archive at NCAR
  • Model output stored as forecast files for each
    data provider
  • Each file contains all parameters and ensemble
    members for a level type and forecast time
  • 2 3 week period of most recent forecast files
    online for direct user download.
  • Complete archive maintained on the Mass Store
    System as part of the CISL Research Data Archive
    (RDA).
  • TIGGE Archive at ECMWF
  • Data archived through ECMWFs MARS system.
  • Stores individual GRIB messages.

7
Archive Center Data Collection
  • Summary of Data Providers

8
Archive Center Data Collection
  • Summary of Data Provider Forecasts

9
Archive Center Data Collection
  • Pressure Level Parameters

10
Archive Center Data Collection
  • Isentropic Level Parameters

Potential Vorticity Level Parameters
11
Archive Center Data Collection
  • Single Level Parameters

12
Archive Center Data Collection
  • Single Level Parameters cont.

13
Archive Center Data Collection
  • Single Level Parameters cont.

14
Archive Center Data Collection
  • Single Level Parameters cont.

15
Archive Center Data Collection
  • Single Level Parameters cont.

16
Data Discovery, Access, and Distribution
  • NCARs TIGGE Web Portal (http//tigge.ucar.edu)
  • Users can search, discover, and download forecast
    files
  • Select data by initialization date/time, data
    provider, parameter level, and forecast time.
  • Each file contains all parameters and ensemble
    members for a given level type
  • Single Level Parameters (Surface)
  • Pressure Level Parameters
  • Potential Vorticity Level Parameters
  • Potential Temperature Level Parameters.
  • Contains most recent 2-3 weeks of model output.
  • 48 hour data access delay from model init time.

17
Data Discovery Access and Distribution
18
Data Discovery Access and Distribution
19
Data Discovery Access and Distribution
  • NCAR Research Data Archive
  • http//dss.ucar.edu/datasets/ds330.0/ -2006
  • http//dss.ucar.edu/datasets/ds330.1/ -2007
  • The RDA dataset enables
  • Access to the complete TIGGE archive.
  • Easy use of the TIGGE archive on NCAR CISL
    computers data coming directly from the Mass
    Store System (MSS)
  • Support staff to handle requests for offline data
  • Forecast file structure identical to online files

20
Data Discovery Access and Distribution
21
Data Discovery Access and Distribution
22
Data Discovery Access and Distribution
23
Data Discovery Access and Distribution
24
Analysis Tools
  • User Analysis and Basic Data Manipulation Tools
  • WMO GRIB2 is relatively new, so tools are
    immature
  • Forecasts with ensemble members add another
    dimension
  • Improvements underway in
  • NCAR Command Language (NCL)/Python
  • Gempak, GRIB-Java (Unidata)
  • NCEP tools (wgrib2, etc…)
  • ECMWF GRIB API
  • http//tigge.ucar.edu/data-tools.htm

25
Future User Services
  • TIGGE Portal 2007 Improvements
  • Develop streaming download for multiple files
  • Upgrade to handle subset data requests through a
    simple interface (e.g. parameter selection)
  • Provide user selected grid interpolation across
    multiple models
  • Add spatial sub-setting functionality
  • Include subscription services for recurring
    requests
  • Provide web services for automated requests
  • Common interface for NCAR, ECMWF, and CMA

26
Summary
  • ECMWF and NCAR now archiving data from 4
    providers (ECMWF, NCEP, UKMO, JMA)
  • 6 more data providers plan to come online over
    2007 (Australia, Brazil, Canada, China, Korea,
    France)
  • The TIGGE archive and access system designed to
    accommodate irregularity between providers.
  • Desired 2007 Improvements include
  • Streaming download for multiple forecast files.
  • User specified sub-set/grid interpolation
    requests across multiple models.
  • A common web service interface for CMA, ECMWF,
    and NCAR for automated requests.

27
TIGGE Portal / Web Services Architecture
  • Java 5 based implementation.
  • Developed with the Spring application framework.
  • Web Portal Interface
  • Point and click interface allows the selection
    and downloading of forecast dataset.
  • Web Services
  • REST based Web Services to allow clients to
    discover and download data.
  • Provides an interface for automated clients to
    download data in large volumes and at regular
    intervals.

28
REST Web Services
  • REST Representational State Transfer.
  • REST is not a standard for web services, it is a
    style of web service.
  • REST is built on existing standards
    (HTTP/URL/XML).
  • Requests are sent as valid HTTP requests using
    the appropriate verbs (GET, POST, PUT, DELETE).
  • Responses are returned service specific XML
    documents.
  • Since REST is not a standard it avoids some of
    the pitfalls and problems associated with SOAP
    based web service interoperability.
  • In this particular application it allows for
    returning complex data types that are not
    inherently bound to implementation
    toolkit/language.

29
TIGGE Data Selection/Download Architecture
Tomcat Engine
MySql
TIGGE Forecast Metadata Database
Forecast Metadate Database Access
Browser Client
TIGGE Web Portal Interface
HTTP Request/Response
TIGGE Forecast Catalog Service
Web Service Client
TIGGE Web Service Interface
HTTP/XML Rest Request/Response
GRIB2 Forecast Data Files
30
TIGGE Data Subsetting Architecture
Tomcat Engine
MySql
TIGGE Subset Request Database
Browser Client
TIGGE Web Portal Interface
Forecast Metadate Database Access
HTTP Request/Response
TIGGE Forecast Catalog Service
Web Service Client
TIGGE Web Service Interface
HTTP/XML Rest Request/Response
Perl Server
Embedded HTTP Client
Data Extraction Engine
31
Archive Center Data Collection
  • Current TIGGE Data Ingest Status
  • Data is coming from three separate IDD/LDM
    systems
  • ECMWF (Data Provider ECMWF, UKMO, JMA)
  • Overwhelmingly the largest
  • CONDUIT (NCEP), maintained by Unidata
  • CPTEC (Brazil)

ECMWF (UKMO)
JMA
CPTEC
NCEP
Unidata Server
NCAR
32
Archive Center Data Collection
Hourly Volume Receipt from IDD EXP Stream
33
Archive Center Data Collection
Hourly Volume Receipt from IDD CONDUIT Stream
34
Archive Center Data Collection
  • Hourly Cumulative Volume Data Summary

35
Archive Center Data Collection
  • Highlights
  • Succeeded to reach target data rate (10 GB/hr)
    Oct. 2005
  • Build up supporting systems and software around
    IDD/LDM
  • Develop archiving procedures and implement file
    management for the TIGGE portal.
  • Begin realistic daily test data flow April 2006
  • Initiate operational data collection October 1,
    2006

36
Future User Services
  • Sub-setting and Grid Interpolation
  • Users will need more than file downloads at model
    center native resolution
  • Specify parameter, temporal, and spatial
    sub-setting across all models
  • Requires verified software from each data
    provider
  • Must build and run effectively on local hardware

37
TIGGE - Who, What, Why?
  • WMO THORPEX Interactive Grand Global Ensemble
    (TIGGE)
  • Foster multi-model ensemble studies.
  • Improve accuracy of 1 to 14 day high-impact
    weather forecasts.
  • Up to 10 operational centers contributing
    ensemble data.
  • Two Phases
  • 1 Three central Archive Centers
  • China Meteorological Administration (CMA)
  • European Center for Medium-Range Weather
    Forecasts (ECMWF)
  • National Center for Atmospheric Research (NCAR)
  • 2 Widely Distributed Access (not discussed
    today)

38
Archive Center Data Collection
  • Cooperative Support Team for Data Ingest and
    Highlights
  • ECMWF Raoult, Fuentes
  • Optimally tune ECWMF and NCAR systems to work
    together
  • Develop protocols to ensure complete transfer
    (e.g. grid manifests)
  • Unidata Yoksas
  • Expert advise on IDD/LDM
  • VETS Brown
  • Installed, configured/re-configured, monitored
    IDD/LDM
  • DSG Arnold
  • System administration
  • NETS Mitchell
  • TCP packet analysis and advise

39
Current and Future Challenges
  • Transition from Dataportal to Ultra-zone was not
    seamless
  • Machine differences perturbed a number of
    adjustments
  • Clock synchronization with ECMWF
  • TCP Cache, and File IO settings
  • LDM configuration
  • Things are running well now.

40
TIGGE Status _at_ NCAR
  • Data Received
  • Data transport by Unidatas IDD/LDM application
  • Tuned to a maximum of about 10 GB/Hour (optimum
    rate between ECMWF and NCAR)
  • Transfer packets are individual GRIB2 messages
    (single 2D fields)
  • Current receipt metrics, 172 GB/Day, 809K
    fields
  • Data organization and storage
  • Fields are combined into forecast files organized
    by
  • Initialization time, data provider, level type,
    and forecast time step
  • All ensemble members are included in each
    forecast file
  • Most current three-week period kept online
    (4-6TB)
  • Long-term times series archived on the NCAR Mass
    Storage System
  • This file based approach contrasts with the
    ECMWF MARS approach

41
TIGGE Status _at_ NCAR
  • Data Providers
  • Current participation
  • N200, Reduced Gaussian
  • There are 71 standard grids, varying levels of
    compliance now
  • Grids 26 surface or single level 45 pressure
    level
  • Future contributors
  • Australia, Brazil, Canada, China, Korea, and
    France
  • Archive System is designed to handle varying
    forecasts per day, output resolution, and
    numbers of ensemble members

42
TIGGE Status _at_ NCAR
  • Next Steps
  • Design and implement user registration
  • Simple online form, agree to research and
    education work only
  • Default, 48 hour delay (from forecast init.
    time) before access
  • Real-time access granted by International
    Program Office
  • E.g. Special field project support
  • Open user access
  • Initially, only file download through a portal
    interface
  • http//tigge.ucar.edu (to be activated early
    November)
  • User Software, a challenge because GRIB2 is new
  • Put examples and resource pointers into portal
    (end November)
  • Improvements will be posted as they become
    available
  • User defined subsets and regridding across
    multiple models
  • Development depends on securing additional
    funding
  • Bring new Data Providers online as they become
    ready
About PowerShow.com