NOAAs National Weather Service Accessing the GRIB2 encoded National Digital Forecast Database

1
NOAAsNational Weather ServiceAccessing the
GRIB2 encoded National Digital Forecast Database

• Arthur Taylor
• NOAA/NWS/Meteorological Development Laboratory
• NOAA WebShop 2004
• July 28, 2004

2
NDFD What is it?

• The National Digital Forecast Database (NDFD), is
  a mosaic of the National Weather Services (NWS)
  digital forecasts
• It is available free to all.
• It is updated hourly with the best forecast
  available to the NWS.
• It contains the following variables at various
  temporal steps out to 7 days
• Maximum Temperature, Minimum Temperature,
  Probability of Precipitation, Temperature, Wind
  Direction, Wind Speed, Dew Point Temperature, Sky
  Coverage, Quantity of Precipitation, Snow Amount,
  Wave Height, and Weather (which includes
  visibility).
• Advantage?
• The NDFD provides a single place to go in order
  to get the best forecast available from the NWS
  each hour.

3
NDFD How do I get it?

• The NDFD is available in the World Meteorological
  Organizations (WMO) GRIB (version 2) format at
• http//weather.noaa.gov/pub/SL.us008001/ST.expr/DF
  .gr2/DC.ndfd/
• ftp//tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr
  2/DC.ndfd/
• It is broken up into various geographic areas
  which include
• The Continental U.S. (CONUS),
• Various predefined sectors which are Pacific
  North West, Pacific South West, Northern Rockies,
  Central Rockies, Southern Rockies, Northern
  Plains, Central Plains, Southern Plains, Upper
  Mississippi Valley, Central Mississippi Valley,
  Southern Mississippi Valley, Central Great Lakes,
  Eastern Great Lakes, North East, South East, and
  Mid Atlantic sectors.
• Puerto Rico.
• Inside a sector it is broken up into the various
  weather elements.

4
NDFD Which Sector?
5
NDFD Custom Sector?

• The predefined sectors are created hourly, and
  are already waiting to be downloaded.
• Advantage Speed.
• Data is already available and ready to transmit
  when the request for the data arrives on the
  server.
• Disadvantage Flexibility.
• If a user is interested in a specific area,
  he/she will either need to download multiple
  sectors, or download excess information.
• Alternate solution?
• Provide a web service which allows the user to
  specify the area they are interested in. The
  service encodes just that portion of the NDFD in
  GRIB2 and sends it back.

6
NDFD Custom Sector.

• The experimental custom sector web service is
  available at
• http//ndfd.weather.gov/cgi-bin/ndfd/gribcut
• The following call gets the maximum temperature
  in a subgrid which is defined with its lower left
  corner at lat30, lon-100, and its upper right
  corner at lat40, lon-90
• gribcut?varmaxtlat130lon1-100lat240lon2-9
  0
• For more information see
• http//www.nws.noaa.gov/ndfd/grib.htm
• Advantage Flexible.
• The user can choose any latitude longitude box.
  Note the box is not allowed to be larger than 15
  degrees on a side.
• Disadvantage Speed.
• The data is encoded upon request, which means
  some work on the server side.
• In a high demand period this method may slow
  down.
• If this becomes an issue, various methods can be
  used to speed it up.

7
NDFD How do I use it?

• Since NDFD uses the WMOs GRIB2 format, how do I
  read a GRIB2 file?
• The NWS has provided NDFD users a program
  (degrib) to easily download and locally decode
  GRIB2 files see
• http//www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/
• degrib can be installed on most local operating
  systems (Windows, Linux, UNIX).
• It provides a graphical user interface for
  specifying products to download and decode, along
  with some image generation capability.
• The software is based on the GRIB2 library
  available at
• http//www.nws.noaa.gov/mdl/iwt/grib2/decoder.htm
• http//www.nws.noaa.gov/mdl/iwt/grib2/encoder.htm

8
Degrib What can it do?

• Download NDFD Data
• Inventory GRIB and GRIB2 files
• Convert GRIB and GRIB2 data to other formats
• ESRI .shp (point, small polygon, large polygon)
  files
• .flt files (used by GrADS and ESRI Spatial
  Analyst)
• netCDF files
• .csv files
• Probe GRIB and GRIB2 data at a given point.
• Interpret the meta-data provided in the GRIB and
  GRIB2 files.
• Note Both a Graphical User Interface (GUI) and
  command line interface are provided. See
  c/ndfd/degrib15/docs/degrib.txt for
  information on the command line interface.

9
Degrib Installation

• Degrib can be downloaded from
• http//www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/regi
  ster.htm
• On the download page, MS-Windows users should get
  ndfd-demo.exe, which contains
• An installation wizard
• Compiled code
• Source code for the libraries, and the degrib
  and tkdegrib programs (superImageGen and
  htmlmaker source code are not provided)
• Documentation
• Unix users should get degrib-unix.tar.gz, which
  contains
• Source code for the libraries and the degrib
  and tkdegrib programs
• Documentation.
• To compile the code you need
• FORTRAN (g77 should work) for the decoder
  library
• C (gcc should work) for the drivers and
  projection library
• Optional Tcl/Tk (free from http//scriptics.com)
  for the Graphical User Interface

10
Degrib Data download

• Step 1 Download some data.
• The GUI version of degrib, tkdegrib, provides
  an interface to get NDFD either via ftp or http.
• Highlight the desired sector
• Click on either Download by ftp or Download by
  http
• Alternatively, you can get the data directly
  from
• http//weather.noaa.gov/pub/SL.us008001/ST.expr/DF
  .gr2/DC.ndfd/
• ftp//tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr
  2/DC.ndfd/

11
Degrib Data download

• Step 1 continued
• The GUI version of degrib now provides a custom
  sector, which you control by
• Use the file-gtconfigure menu.
• Choose the Custom tab.
• Enter in the desired area, press ok.
• In addition there is a command line method to
  download various sectors and variables, using
• cd C\ndfd\degrib15\bin\
• tcldegrib web.tcl

12
Degrib Convert to .shp file

• To convert to .shp format
• Double click on the GRIB2 file (top pane)
• Select message to convert (middle pane)
• Choose an output file name
• Click on Generate .shp file
• Small Polygon creates better images in ESRI and
  is easier to manipulate, but it is larger than
  the Point .shp
• Large Polygon merges polygons together to
  create smaller files, but makes grid to grid
  comparison more difficult.
• Include Missing Values allows you to study the
  entire grid, or limit the result to where data
  exists

13
CONUS MaxT in ArcView
After converting the grid, start up ESRI ArcView,
then load the layer (.shp file) that you just
created. Use the color legend found
in C/ndfd/degrib15/arcview/poly_legend (for
polygon shapefiles) C/ndfd/degrib15/arcview/point
_legend (for point shapefiles)
14
Polygon vs Point, DC area
15
NDFD CONUS Map Projection

• For CONUS (continental U.S.), NDFD uses a Lambert
  Conformal Conic Map Projection
• Tangent latitutude 25, orientation longitude -95,
  mesh latitude 25, mesh size 5079.406 m.
• In ESRI ArcMap that would be
• 1st Parallel 2nd Parallel 25 degrees north
  (tangent latitude)
• Lon of center of projection -95 degrees
  (orientation longitude)
• Lat of center of projection 25 degrees (tangent
  latitude)
• False easting 0 m, False northing 0 m, X
  Shift 0, Y Shift 0
• In addition, NDFD uses a spherical earth with
  radius 6371.2 km
• To assist ESRI ArcView 3.x, degrib creates a .ave
  (ArcView Avenue script) to set the projection and
  radius correctly.
• For ArcGIS, please see /degrib/arcview/prj.adf,
  and ndfd.prj

16
Un-projected vs Projected
17
Degrib Convert to .flt file

• To convert to .flt file (for use with GrADS or
  ESRI Spatial Analyst)
• Double click on the GRIB2 file (top pane)
• Select message to convert (middle pane)
• Choose an output file name
• Click on Generate .flt file
• Grid allows you to select what kind of
  interpolation to do (or keep the original
  projection).
• GrADS .ctl file creates a control file which
  GrADS can use to access the .flt file
• M.S.B. first creates the .flt files in Big
  Endian format.

18
CONUS MaxT in GrADS
19
Degrib Convert to netCDF file

• To convert to netCDF file
• Double click on the GRIB2 file (top pane)
• Select message to convert (middle pane)
• Choose an output file name
• Click on Generate NetCDF (.nc) file
• This creates a netCDF file according to the CF
  metadata conventions.
• This converts 1 GRIB message to 1 netCDF file.
• An improvement would be for multiple GRIB
  messages to go to 1 netCDF file.

20
CONUS MinT in NetCDF
21
Degrib Convert to .csv file

• To convert to CSV file
• Double click on the GRIB2 file (top pane)
• Select message to convert (middle pane)
• Choose an output file name
• Click on Generate .csv files
• This creates a comma separated text file which
  can be worked with in excel.
• Note Excel has problems loading the CONUS NDFD,
  as it appears to be limited to 65,536 records,
  while the NDFD typically has 739,297 records

22
CONUS MinT in .csv in Excel
23
Degrib Probe point

• Degrib also provides the ability to probe a GRIB
  file at a set of latitude longitude locations.
  It computes the value at that spot, and returns a
  text message. Note this is not available in the
  GUI. To do so from the command line
• C\ndfd\degrib\bin\degrib maxt.bin P pnt
  38.99,-77.03
• C\ndfd\degrib\bin\degrib maxt.bin P pntFile
  point.txt
• The first example probes just 1 point, while the
  second reads a file which is of the form
  StationID, lat, lon per line. The second form
  allows for numerous stations.
• By default, probe chooses the nearest grid
  cell, but using the -Interp option, causes it
  to perform bi-linear interpolation.
• See (/degrib/docs/degrib.txt for more help)

24
Probe point (MaxT, MinT, PoP)
element, unit, refTime, validTime,
(38.993600,-77.022400) PoP12, , 200307212300,
200307220000, 39.000 PoP12, , 200307212300,
200307221200, 39.000 PoP12, , 200307212300,
200307230000, 50.000 PoP12, , 200307212300,
200307231200, 50.000 PoP12, , 200307212300,
200307240000, 50.000 PoP12, , 200307212300,
200307241200, 50.000 PoP12, , 200307212300,
200307250000, 20.000 PoP12, , 200307212300,
200307251200, 20.000 PoP12, , 200307212300,
200307260000, 14.000 PoP12, , 200307212300,
200307261200, 5.000 PoP12, , 200307212300,
200307270000, 5.000 PoP12, , 200307212300,
200307271200, 11.000 PoP12, , 200307212300,
200307280000, 11.000 PoP12, , 200307212300,
200307281200, 21.000 PoP12, , 200307212300,
200307290000, 21.000
element, unit, refTime, validTime,
(38.993600,-77.022400) MaxT, F, 200307212300,
200307230000, 86.882 MaxT, F, 200307212300,
200307240000, 82.375 MaxT, F, 200307212300,
200307250000, 85.233 MaxT, F, 200307212300,
200307260000, 87.753 MaxT, F, 200307212300,
200307270000, 89.395 MaxT, F, 200307212300,
200307280000, 87.775 MaxT, F, 200307212300,
200307290000, 86.800 element, unit, refTime,
validTime, (38.993600,-77.022400) MinT, F,
200307212300, 200307221200, 72.625 MinT, F,
200307212300, 200307231200, 73.165 MinT, F,
200307212300, 200307241200, 69.025 MinT, F,
200307212300, 200307251200, 65.965 MinT, F,
200307212300, 200307261200, 67.945 MinT, F,
200307212300, 200307271200, 69.745 MinT, F,
200307212300, 200307281200, 69.025
25
NDFD, GRIB2 Weather

• To encode Weather (Wx), NDFD disseminates
  weather strings.
• Advantage A flexible format that can encode a
  description of weather such as Chance of
  thunderstorms and chance of heavy rain showers
• Difficulty A weather string is not a number,
  so it is challenging to store in GRIB2.
• Solution Use GRIB2 section 2 to provide an ASCII
  look up table, and store the numeric values in
  the regular GRIB2
• Note each weather grid, for each forecast
  projection, has a different ASCII look up table,
  so without section 2 it has no meaning.
• Some Questions
• What does ChcTltNoIntengtltNoVisgtChcRWltNoVi
  sgt mean, and how, for example, is an Emergency
  Manager supposed to know that?
• What can be done for the .flt file?

26
Degrib Future?

• Maintain the program by keeping up with NDFD as
  more variables / sectors are introduced.
• In order to inform users of updates, degrib has
  an announcement mailing list, which at the last
  release had 434 people on it.
• Improvements Based on user feedback, current
  thoughts are
• Create some form of graphical probe point
  capability
• Add some way for UNIX users (without GrADS) to
  plot the images
• Add some way for users to zoom in on a map
• Improve the NetCDF and GrADS file outputs to
  contain multiple GRIB messages.