Advanced Utilities Extending ncgen to support the netCDF4 Data Model

1
Advanced UtilitiesExtending ncgen to support
the netCDF-4 Data Model
Dr. Dennis Heimbigner Unidata netCDF
Workshop August 3-4, 2009
2
Overview

• The NCGEN4 Utility
• NCGEN4 Command Synopsis
• netCDF4 in CDL Types
• netCDF4 in CDL Typed Attributes
• netCDF4 in CDL Groups
• Scope Rules
• Specifying Data Constants
• Rules for Using Braces
• Special Attributes
• Debugging Note The Cycle
• Building and Installing NCGEN4
• Extended Example

3
The NCGEN4 Utility

• NCGEN4 extends the CDL language
• Includes all of the netCDF-4 data model
• True inverse of ncdump
• Includes the special attributes (chunking, etc)
• Supports binary, C, NOT FORTRAN (yet)
• Experimental Java and NcML
• Can produce 4 kinds of binary netCDF files
• netcdf-3 (classic) 32 bit
• netcdf-3 (classic) 64 bit
• netcdf-3 (classic, but stored in netcdf-4 file
  format)
• netcdf-4 (supports full netcdf-4 model)

4
NCGEN4 Command Synopsis
Primary Command Line Options -b
Create a (binary) netCDF file
(default) -o ltfilegt Name for the
binary netCDF file created -k ltfile_formatgt
Format of the file to be created
1 gt classic 32 bit
2 gt classic 64 bit
3 gt netcdf-4/CDM
4 gt classic, but stored in an enhanced file
format -x Don't
initialize data with fill values -l ltlanguagegt
Specify output language to use when
generating source
code to create or define a netCDF file
matching the CDL specification. ltfilenamegt
Input CDL file
5
netCDF4 in CDL Types

• New section called types
• Consistent with the output of ncdump
• Supports the new primitive data types
• ubyte, ushort, uint, string, int64 (LL),
  uint64 (ULL)
• Supports the new user-defined types
• int enum enum_t off0,on1,unknown2
• opaque(11) opaque_t
• compound cmpd_t vlen_t f1 enum_t f2
• int() vlen_t

6
netCDF4 in CDL Typed Attributes

• vlen_t vattr 17, 18, 19
• Attribute typing is optional (gt type inferred)
• Warning! xattr abc is inferred to be type
  char, not string
• Instead say string xattr abc
• Why? for backward compatibility with ncgen
• Good practice to add _t to the end of all type
  names
• Why? Because X attr might be interpreted
  incorrectly is X a type or variable?

7
netCDF4 in CDL Groups

• group g
• A group can itself contain dimensions, types,
  variables, and groups
• Name prefixing allows references to types and
  dimensions that reside in other groups
• Example /g/cmpd_t
• gt Do not use / in your names
• Pretty much like the Unix file system
• Or Windows, but using forward slashes

8
Scope Rules

• Scope rules determine how references to a
  dimension or type without a prefix are
  interpreted
• General rule
• Look in immediately enclosing group
• Look in the parent of the immediately enclosing
  group and so on up the enclosing groups
• For dimensions, if not found gt error
• For types, continue to search the whole group
  tree to find a unique match, then error if not
  found

9
Specifying Data Constants

• Constants for user defined types require the use
  of braces in certain places.

dimensions d2 types int() vlen_t
compound cmpd_t int64 f1 string
f2 variables vlen_t v1(d) cmpd_t
v2(d) data v1 7, 8, 9, 17,18,19 v2
107LL, abc, 1234567LL, xyz
10
Rules for Using Braces

• The top level is automatically assumed to be a
  list of items, so it should not be inside ...
• Different than C constants lists
• Instances of UNLIMITED dimensions (other than as
  the first dimension) must be surrounded by ...
  in order to specify the size.
• Instances of VLENs must be surrounded by ... in
  order to specify the size.
• Compound instances must be embedded in ...
• Compound fields may optionally be embedded in
  ....
• No other use of braces is allowed.

11
Special Attributes

• Special attributes specified in an ncgen4 CDL
  file will be properly handled
• Consistent with ncdump -s
• Global special attributes
• _Format specify the netCDF file format
• classic
• 64-bit offset
• netCDF-4
• netCDF-4 classic model
• Overridden by the -k flag

12
Special Attributes (cont.)

• Per-variable special attributes
• _ChunkSizes list of chunk sizes 1 per
  dimension
• _DeflateLevel compression level integer
  (0-9)
• _Endianness big or little
• _Fletcher32 true or false to set check
  summing
• _NoFill true or false to set persistent
  NoFill property
• _Shuffle true or false to set shuffle
  filter
• _Storage contiguous or chunked to set
  storage mode

13
Debugging Note

• Use the Cycle, Luke
• Use ncgen/ncgen4 to convert your ltfilegt.cdl to
  ltfilegt.nc
• Then use ncdump to convert your ltfilegt.nc to
  ltfile2gt.cdl
• Compare ltfilegt.cdl to ltfile2gt.cdl
• Watch out for UNLIMITED!

dimensions u unlimited variables v1(u)
v2(u) data v1 1,2,3,4 v2 7,8 Ncdump
produces v2 7,8,_,_
14
Building and Installing NCGEN4

• Easy add --enable-ncgen4 to your list of
  ./configure flags
• Ncgen4 will be installed along with ncdump and
  the original ncgen

15
Extended Example
netcdf foo types ubyte enum enum_t Clear
0, Cumulonimbus 1, Stratus 2
opaque(11) opaque_t int()
vlen_t dimensions lat 10 lon 5 time
unlimited variables long lat(lat),
lon(lon), time(time) float
Z(time,lat,lon), t(time,lat,lon) double
p(time,lat,lon) long rh(time,lat,lon)
string country(time,lat,lon) ubyte
tag
16
Extended Example (cont.)
// variable attributes latlong_name
"latitude" latunits "degrees_north"
lonlong_name "longitude" lonunits
"degrees_east" timeunits "seconds since
1992-1-1 000000" // typed variable
attributes string Zunits "geopotential
meters" float Zvalid_range 0.,
5000. double p_FillValue -9999. long
rh_FillValue -1 vlen_t globalatt
17, 18, 19
17
Extended Example (cont.)
data lat 0, 10, 20, 30, 40, 50, 60, 70,
80, 90 lon -140, -118, -96, -84,
-52 group g types compound
cmpd_t vlen_t f1 enum_t f2 // group
g group h variables /g/cmpd_t
compoundvar data compoundvar
3,4,5, Stratus // group h
18
Questions?