NetCDF

1
NetCDF

• Ed Hartnett
• Unidata/UCAR
• ed_at_unidata.ucar.edu

2
Unidata

• Unidata - helps universities acquire, display,
  and analyze Earth-system data.
• UCAR University Corporation for Atmospheric
  Research - a nonprofit consortium of 66
  universities.

3
SDSC Presentation, July 2005

• Intro to NetCDF Classic
• Intro to NetCDF-4

4
What is NetCDF?

• A conceptual data model for scientific data.
• A set of APIs in C, F77, F90, Java, etc. to
  create and manipulate data files.
• Some portable binary formats.
• Useful for storing arrays of data and
  accompanying metadata.

5
History of NetCDF
netCDF 4.0 beta released
netCDF developed at Unidata
netCDF 3.0 released
2005
1988
2004
1991
1996
netCDF 2.0 released
netCDF 3.6.0 released
6
Getting netCDF

• Download latest release from the netCDF web page
  http//www.unidata.ucar.edu/content/software/netcd
  f
• Builds and installs on most platforms with no
  configuration necessary.
• For a list platforms netCDF versions have built
  on, and the output of building and testing
  netCDF, see the web site.

7
NetCDF Portability

• NetCDF is tested on a wide variety of platforms,
  including Linux, AIX, SunOS, MacOS, IRIX, OSF1,
  Cygwin, and Windows.
• We test with native compilers when we can get
  them.
• 64-bit builds are supported with some
  configuration effort.

8
What Comes with NetCDF

• NetCDF comes with 4 language APIs C, C,
  Fortran 77, and Fortran 90.
• Tools ncgen and ncdump.
• Tests.
• Documentation.

9
NetCDF Java API

• The netCDF Java API is entirely separate from the
  C API.
• You dont need to install the C API for the Java
  API to work.
• Java API contains many exciting features, such as
  remote access and more advanced coordinate
  systems.

10
Tools to work with NetCDF Data

• The netCDF core library provides basic data
  access.
• ncgen and ncdump provide some helpful command
  line functionality.
• Many additional tools are available, see
  http//www.unidata.ucar.edu/packages/netcdf/softwa
  re.html

11
CDL Common Data Language

• Grammar defined for displaying information about
  netCDF files.
• Can be used to create files without programming.
• Can be used to create reading program in Fortran
  or C.
• Used by ncgen/ncdump utilities.

12
Example of CDL

• netcdf foo // example netCDF specification in
  CDL
• dimensions
• lat 10, lon 5, time unlimited
• variables
• int lat(lat), lon(lon), time(time)
• float z(time,lat,lon), t(time,lat,lon)
• double p(time,lat,lon) int rh(time,lat,lon)
• latunits "degrees_north"
• lonunits "degrees_east"
• data
• lat 0, 10, 20, 30, 40, 50, 60, 70, 80, 90
• lon -140, -118, -96, -84, -52

13
Software Architecture of NetCDF-3
V2 C tests
F77 tests
F90 API
V2 C API
V3 C tests
F77 API
C API
ncdump
ncgen
V3 C API

• Fortran, C and V2 APIs are all built on the C
  API.
• Other language APIs (perl, python, MatLab, etc.)
  use the C API.

14
NetCDF Documentation

• Unidata distributes a NetCDF Users Guide which
  describes the data model in detail.
• A language-specific guide is provided for C, C,
  Fortran 77, and Fortran 90 users.
• All documentation can be found at
  http//my.unidata.ucar.edu/content/software/netcdf
  /docs

15
NetCDF Jargon

• Variable a multi-dimensional array of data,
  of any of 6 types (char, byte, short, int, float,
  or double).
• Dimension information about an axis its
  name and length.
• Attribute a 1D array of metadata.

16
More NetCDF Jargon

• Coordinate Variable a 1D variable with the
  same name as a dimension, which stores values for
  each dimension value.
• Unlimited Dimension a dimension which has no
  maximum size. Data can always be extended along
  the unlimited dimension.

17
The NetCDF Classic Data Model

• The netCDF Classic Data Model contains
  dimensions, variables, and attributes.
• At most one dimension may be unlimited.
• The Classic Data Model is embodied by netCDF
  versions 1 through 3.6.0
• NetCDF is moving towards a new, richer data
  model the Common Data Model.

18
NetCDF Example

• Suppose a user wants to store temperature and
  pressure values on a 2D latitude/longitude grid.
• In addition to the data, the user wants to store
  information about the lat/lon grid.
• The user may have additional data to store, for
  example the units of the data values.

19
NetCDF Model Example
Variables
Dimensions
Attributes
temperature
latitude
Units C
pressure
longitude
Units mb
Coordinate Variables
latitude
longitude
20
Important NetCDF Functions

• nc_create and nc_open to create and open files.
• nc_enddef, nc_close.
• nc_def_dim, nc_def_var, nc_put_att_, to define
  dimensions, variables, and attributes.
• nc_inq, nc_inq_var, nc_inq_dim, nc_get_att_ to
  learn about dims, vars, and atts.
• nc_put_vara_, nc_get_vara_ to write and read
  data.

21
C Functions to Define Metadata

• / Create the file. /
• if ((retval nc_create(FILE_NAME, NC_CLOBBER,
  ncid)))
• return retval
• / Define the dimensions. /
• if ((retval nc_def_dim(ncid, LAT_NAME,
  LAT_LEN, lat_dimid)))
• return retval
• if ((retval nc_def_dim(ncid, LON_NAME,
  LON_LEN, lon_dimid)))
• return retval
• / Define the variables. /
• dimids0 lat_dimid
• dimids1 lon_dimid
• if ((retval nc_def_var(ncid, PRES_NAME,
  NC_FLOAT, NDIMS, dimids, pres_varid)))
• return retval
• if ((retval nc_def_var(ncid, TEMP_NAME,
  NC_FLOAT, NDIMS, dimids, temp_varid)))
• return retval
• / End define mode. /

22
C Functions to Write Data

• / Write the data. /
• if ((retval nc_put_var_float(ncid,
  pres_varid, pres_out)))
• return retval
• if ((retval nc_put_var_float(ncid,
  temp_varid, temp_out)))
• return retval
• / Close the file. /
• if ((retval nc_close(ncid)))
• return retval

23
C Example Getting Data

• / Open the file. /
• if ((retval nc_open(FILE_NAME, 0, ncid)))
• return retval
• / Read the data. /
• if ((retval nc_get_var_float(ncid, 0,
  pres_in)))
• return retval
• if ((retval nc_get_var_float(ncid, 1,
  temp_in)))
• return retval
• / Do something useful with the data /
• / Close the file. /
• if ((retval nc_close(ncid)))
• return retval

24
Data Reading and Writing Functions

• There are 5 ways to read/write data of each type.
• var1 reads/writes a single value.
• var reads/writes entire variable at once.
• vara reads/writes an array subset.
• vars reads/writes an array by slices.
• varm reads/writes a mapped array.
• Ex. nc_put_vars_short writes shorts by slices.

25
Attributes

• Attributes are 1-D arrays of any of the 6 netCDF
  types.
• Read/write them with functions like
  nc_get_att_float and nc_put_att_int.
• Attributes may be attached to a variable, or may
  be global to the file.

26
NetCDF File Formats

• Starting with 3.6.0, netCDF supports two binary
  data formats.
• NetCDF Classic Format is the format that has been
  in use for netCDF files from the beginning.
• NetCDF 64-bit Offset Format was introduced in
  3.6.0 and allows much larger files.
• Use classic format unless you need the large
  files.

27
NetCDF-3 Summary

• NetCDF is a software library and some binary data
  formats, useful for scientific data, developed at
  Unidata.
• NetCDF organizes data into variables, with
  dimensions and attributes.
• NetCDF has proven to be reliable, simple to use,
  and very popular.

28
Why Add to NetCDF-3?

• Increasingly complex data sets call for greater
  organization.
• Size limits, unthinkably huge in 1988, are
  routinely reached in 2005.
• Parallel I/O is required for advanced Earth
  science applications.
• Interoperability with HDF5.

29
NetCDF-4

• NetCDF-4 aims to provide the netCDF API as a
  front end for HDF5.
• Funded by NASA, executed at Unidata and NCSA.
• Includes reliable netCDF-3 code, and is fully
  backward compatible.

30
NetCDF-4 Organizations

• Unidata/UCAR
• NCSA The National Center for Supercomputing
  ApplicationsUniversity of Illinois at
  Urbana-Champaign
• NASA NetCDF-4 was funded by NASA award number
  AIST-02-0071.

31
New Features of NetCDF-4

• Multiple unlimited dimensions.
• Groups to organize data.
• New types, including compound types and variable
  length arrays.
• Parallel I/O.

32
The Common Data Model

• NetCDF-4, scheduled for beta-release this Summer,
  will conform to the Common Data Model.
• Developed by John Caron at Unidata, with the
  cooperation of HDF, OpenDAP, netCDF, and other
  software teams, CDM unites different models into
  a common framework.
• CDM is a superset of the NetCDF Classic Data Model

33
The NetCDF-4 Data Model

• NetCDF-4 implements the Common Data Model.
• Adds groups, each group can contain variables,
  attributes and dimensions, and groups.
• Dimensions are scoped so that variables in
  different groups can share dimensions.
• Compound types allow users to define new types,
  comprised of other atomic or user-defined types.
• New integer and string types.

34
Software Architecture of NetCDF-4
V2 C tests
F77 tests
F90 API
V2 C API
V3 C tests
F77 API
C API
ncdump
ncgen
V4 C API
V3 C API
HDF5
35
NetCDF-4 Release Status

• Latest alpha release includes all netCDF-4
  features depends on latest HDF5 development
  snapshot.
• Beta release due out in August, replaces
  artificial netCDF-4 constructs, and depends on a
  yet-to-be-released version of HDF5.
• Promotion from beta to full release will happen
  sometime in 2006.

36
Building NetCDF-4

• NetCDF-4 requires that HDF5 version 1.8.3 be
  installed. This is not released yet.
• The latest HDF5 development release works with
  the latest netCDF alpha release.
• To build netCDF-4, specify enable-netcdf-4 at
  configure.

37
When to Use NetCDF-4 Format

• The new netCDF-4 features (groups, new types,
  parallel I/O) are only available for netCDF-4
  format files.
• When you need HDF5 files.
• When portability is less important, until
  netCDF-4 becomes widespread.

38
Versions and Formats
netCDF developed by Glenn Davis
netCDF 4.0 beta released
netCDF 3.0 released
2005
1988
2004
1991
1996
netCDF 2.0 released
netCDF 3.6.0 released
NetCDF-4 Format
64-Bit Offset Format
Classic Format
39
NetCDF-4 Feature Review

• Multiple unlimited dimensions.
• How to use groups.
• Using compound types.
• Other new types.
• Variable length arrays.
• Parallel I/O.
• HDF5 Interoperability.

40
Multiple Unlimited Dimensions

• Unlimited dimensions are automatically expanded
  as new data are written.
• NetCDF-4 allows multiple unlimited dimensions.

41
Working with Groups

• Define a group, then use it as a container for
  the classic data model.
• Groups can be used to organize sets of data.

42
An Example of Groups
43
New Functions to Use Groups

• Open/create returns ncid of root group.
• Create a new group with nc_def_grp.
• nc_def_grp(int parent_ncid, char name, int
  new_ncid)
• Learn about groups with nc_inq_grps.
• nc_inq_grps(int ncid, int numgrps, int ncids)

44
C Example Using Groups

• if (nc_create(FILE_NAME, NC_NETCDF4, ncid))
  ERR
• if (nc_def_grp(ncid, DYNASTY, tudor_id))
  ERR
• if (nc_def_dim(tudor_id, DIM1_NAME,
  NC_UNLIMITED, dimid)) ERR
• if (nc_def_grp(tudor_id, HENRY_VII,
  henry_vii_id)) ERR
• if (nc_def_var(henry_vii_id, VAR1_NAME,
  NC_INT, 1, dimid, varid)) ERR
• if (nc_put_vara_int(henry_vii_id, varid,
  start, count, data_out)) ERR
• if (nc_close(ncid)) ERR

45
Create Complex Types

• Like C structs, compound types can be assembled
  into a user defined type.
• Compound types can be nested that is, they can
  contain other compound types.
• New functions are needed to create new types.
• V2 API functions are used to read/write complex
  types.

46
C Example of Compound Types

• / Create a file with a compound type. Write a
  little data. /
• if (nc_create(FILE_NAME, NC_NETCDF4,
  ncid)) ERR
• if (nc_def_compound(ncid, sizeof(struct
  s1), SVC_REC, typeid)) ERR
• if (nc_insert_compound(ncid, typeid,
  BATTLES_WITH_KLINGONS,
• HOFFSET(struct s1, i1), NC_INT)) ERR
• if (nc_insert_compound(ncid, typeid,
  DATES_WITH_ALIENS,
• HOFFSET(struct s1, i2), NC_INT)) ERR
• if (nc_def_dim(ncid, STARDATE, DIM_LEN,
  dimid)) ERR
• if (nc_def_var(ncid, SERVICE_RECORD,
  typeid, 1, dimids, varid)) ERR
• if (nc_put_var(ncid, varid, data)) ERR
• if (nc_close(ncid)) ERR

47
New Ints, Opaque, String Types

• Opaque types are bit-blobs of fixed size.
• String types allow multi-dimensional arrays of
  strings.
• New integer types UBYTE, USHORT, UINT, UINT64,
  INT64.

48
Variable Length Arrays

• Variable length arrays allow the efficient
  storage of arrays of variable size.
• For example an array of soundings of different
  number of elements.

49
Parallel I/O with NetCDF-4

• Must use configure option enable-parallel when
  building netCDF.
• Depends on HDF5 parallel features, which require
  MPI.
• Must create or open file with nc_create_par or
  nc_open_par.
• All metadata operations are collective.
• Adding a new record is collective.
• Variable reads/writes are independent by default,
  but can be changed to do collective operations.

50
HDF5 Interoperability

• NetCDF-4 can interoperate with HDF5 with a SUBSET
  of HDF5 features.
• Will not work with HDF5 files that have looping
  groups, references, and types not found in
  netCDF-4.
• HDF5 file must use new dimension scale API to
  store shared dimension info.
• If a HDF5 follows the Common Data Model, NetCDF-4
  can interoperate on the same files.

51
Future Plans for NetCDF

• NetCDF 4.0 release in 2006.
• Beta for next major version of netCDF in Summer,
  2006.
• Full compatibility with Common Data Model.
• Remote access, including remote subsetting of
  data.
• XML-based representation of netCDF metadata.
• Full Fortran 90 support, but limited F77 support.

52
For Further Information

• netCDF mailing list netcdfgroup_at_unidata.ucar.edu
  
• email Ed ed_at_unidata.ucar.edu
• netCDF web site www.unidata.ucar.edu