Support for NPP/NPOESS by The HDF Group

1
Support for NPP/NPOESS by The HDF Group

• Mike Folk, Elena Pourmal, Peter Cao
• The HDF Group
• November 5, 2009

2
Goal

• To develop and adapt HDF5 software and provide
  HDF5 support for NPP/NPOESS.

3
Areas of immediate need

• Ensure easy and intuitive data access and use by
  diverse communities
• Enable producers and consumers to view content,
  manage metadata, and convert data to other
  formats.
• Provide high-quality, rapid-response support of
  HDF5 for NPOESS users at all levels.

4
2009-2009 Priorities

• Data accessibility and usability
• Developed library of high level APIs to support
  NPP/NPOESS data management
• Tool development
• Modify h5dump to enable viewing of region
  references and quality flags
• Modify HDFView to enhance viewing of references
  and quality flags
• System maintenance
• User support

5
NPP/NPOESS Project Information

• Project Web site
• http//www.hdfgroup.org/projects/npoess/
• Project description
• Released software
• Documentation

6
NPOESS Web site
7
NPOESS Software FTP Site
8
NPOESS Project Information

• Project Wiki site
• http//wiki.hdfgroup.org/NPOESS-Project
• Monthly progress reports, links to software under
  development, presentations, etc.
• Requires a password
• Contact epourmal_at_hdfgroup.org

9
HDF NPOESS Wiki
10
Library of High-Level APIs to support NPOESS
data
11
NPOESS High-Level Library

• Library to support HDF5 region references and
  packed bits in support of NPOESS quality flags
• C APIs
• Documentation
• Tests
• Examples
• Source available from
• ftp//ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/so
  urce/
• Documentation is available from
• http//www.hdfgroup.org/projects/npoess/
• Still a prototype!

12
Documentation
13
h5dump New Features to Support Region
References and Quality Flags
14
h5dump and Region Reference Data

• New flag R to display data pointed to by region
  references
• Improved display of subsets
• An element of a dataset is specified by its
  indices
• Available in HDF5 1.8.4, November 2009

15
h5dump and Quality Flags

• New flag M to display packed bit-field data
• Available only in the HDF5 NPOESS branch
• Will NOT be in HDF5 1.8.4 release
• Source and pre-built binaries can be found at
• ftp//ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/

16
HDFView New Features to Support References and
Quality Flags
17
HDFView and references

• Aggregated granules show the whole dataset
  pointed to by an object reference
• Granules show data pointed to by a dataset
  region reference

18
Objects Pointed to by Object References
19
Data Pointed to by Region References
20
HDFView and quality flags

• By default, integers with quality flags are
  displayed as 8-bit decimal values
• Needed a way to show values of specific quality
  flag bit fields

21
Default display 8-bit decimal integers
22
Displaying quality flags in 8-bit integers

• Use Open As on dataset with quality flags.
• Apply bitmask to choose individual quality flags

23
Examples of bit masks

• Applied mask in red at top-right, e.g., By
  bitmask 0, 1

24
Quality Flags

• Data can be displayed in decimal, binary, or
  hexadecimal

decimal
binary
hexadecimal
25
HDFView plug-in to Show Quality Flags

• Implemented using TreeView plug-in
• Open HDFView
• Select User Options
• Select NPOESS plug-in
• Restart HDFView
• Available with HDFView 2.5 patch3

26
Possible Future Work
27
Possible future work

• Add new features
• Enhance current features
• System support
• User support

28
Maintenance

• Maintenance NPOESS/NPP features
• High-Level Library
• H5dump, other tools?
• HDFView
• Testing, documentation, releases

29
New features

• Support Variable-Length datasets
• Ease access to information about quality flags
• Use definition of quality flags, which are in
• The Common Data Format Control Book
• XML files as part of the CDFCB-X delivery
• Apply the structure of the XML to determine
  whether a field is a packed quality flag byte
• Tell us what you need!

30
Enhance current features

• Extend reference display options
• E.g., show properties of the referenced data
• Revise default TreeView and TableView to improve
  NPOESS plug-in support
• Improve quality of testing, documentation, and
  binary and source distribution
• Tell us what you need!

31
NPOESS/NPP System Support

• Daily testing on systems critical to NPOESS/NPP
• Linux 32 and 64-bit
• Windows
• AIX 5.3
• Solaris 32 and 64-bit
• Regular testing of NPOESS high-Level library
• Linux 32 and 64-bit
• Windows
• Solaris 32 and 64-bit
• Porting to requested compilers
• Performance tuning for applications
• Improved support for C
• Support for netCDF-4 access to NPOESS SDR and EDR
  products

32
NPOESS/NPP User Support

• Priority support to NPOESS/NPP users
• Send email to help_at_hdfgroup.org
• Add NPOESS to the subject field
• Example Subject NPOESS Where is the plug-in?
• Response in less than 4 hours
• Continue gather requirements and enhance HDF5
  software as needed
• We want to hear from you!

33
Special Research Projects

• Improvements to HDF5 or HDF5 NPOESS products with
  the data management portion of the Raytheon
  Algorithm Development Library (ADL)
• Implement OpeNDAP server for HDF5 NPOESS SDR and
  EDR products
• Develop NPOESS specialized version of the HDF5
  tool h5check
• Performance tuning of HDF-NPOESS product files

34
Questions?
35
Questions/comments?
36
Acknowledgements

• This work was supported by cooperative agreement
  number NA08AANEG0128from the National Aeronautics
  and Space Administration (NASA).
• Any opinions, findings, conclusions, or
  recommendations expressed in this material are
  those of the authors and do not necessarily
  reflect the views of the National Aeronautics and
  Space Administration.