HDF5 OPeNDAP Project Update and Demo

1
HDF5 OPeNDAP ProjectUpdate and Demo

• MuQun Yang and Hyo-Kyung Lee
• (The HDF Group)
• James Gallagher
• (OPeNDAP, Inc.)

2
OPeNDAP Review

• A software framework that allows
• simple access to remote data
• Data Access Protocol
• Client-server model
• Widely used by Earth Science
• Community

3
OPeNDAP Concept
Typical Remote Access
Remote Data (HDF4)
View Data (User)
FTP/HTTP
Handler (hdf4_handler)
Visualization Tools (gradsdap)
Visualization tools (e.g. grads)
Client Library (libnc-dap)
Server (apache)
DAP Protocol (via http)
4
OPeNDAP HDF5 Handler
Remote HDF5
View Data (User)
HDF5 Handler
Server (apache)
DAP Protocol (via http)
5
Challenges
Mapping HDF5 to DAP

• Compound Datatype
• Groups
• Object/Regional References

6
OPeNDAP HDF5 Handler with HDF-EOS5
View Data (User)
Remote HDF5
Remote HDF-EOS5
HDF5 Handler
Visualization tools (grads)
Server (apache)
Client Library (libnc-dap)
DAP Protocol (via http)
7
Challenges HDF- EOS5

• Grid with No Geolocation data
• Clients expect Grid with Geolocation data
• Some Attributes stored as Extremely Long
• String(s) (e.g. StructMetada.0)
• Clients expect structured attributes

8
Tweaks for HDF-EOS5

• Added two default HDF5 handler configuration
  options
• --enable-eos-grid
• --enable-eos-meta

9
Challenges OPeNDAP Clients

• Need special attributes on dataset.
• Need shared geolocation variables.

10
Tweaks for OPeNDAP Clients

• Added two optional handler configuration
  options
• --enable-short-name
• --enable-CF

11
OPeNDAP HDF5 Handler with HDF-EOS5
View Data (User)
Remote HDF5
Remote HDF-EOS5
HDF5 Handler
HDF5 Handler w/ CF options
Visualization tools (grads)
Server (apache)
Client Library (libnc-dap)
DAP Protocol (via http)
12
Day After Server Tweaks

• Finally, Happy Clients!

IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
GrADS
ODC
13
Problems of Tweaking HDF5 Handler

• Remember that we added two optional HDF5 handler
  configuration options
• --enable-short-name
• --enable-CF

Causes ambiguity among variable names (e.g.
/GroupA/ozone vs. /GroupB/ozone)
Drops some key attributes (e.g. StructMetdata,
HDF_ROOT_GROUP)
14

• HDF5-Friendly OPeNDAP Client Library

HDF5 Groups
View Groups
???
View Data (User)
Remote HDF5
HDF5 Handler
Visualization tools (grads)
HDF5 Handler w/ CF options
Server (apache)
Client Library (libnc-dap)
HDF5-Friendly Client Library (liboc-dap)
DAP Protocol (via http)
15
Traditional OPeNDAP client library Its an
attribute that I dont understand. Ill ignore it.
Example Groups in HDF5
HDF5-Friendly OPeNDAP client library I was
waiting for this key attribute to re-construct
HDF5
16
Example Reference in HDF5

• Important for NPOESS
• Object / Regional Reference
• Map to DAP URL

17
Example dap2h5
Group/Ref.
View Group/Ref.
View Data (User)
Remote HDF5

• A test application for the client library
• It can construct HDF5 from DAP output

HDF5-F. Client Library App. (dap2h5)
HDF5 Handler
ORIGINAL
CONSTRUCTED
Server (apache)
HDF5-Friendly Client Library (liboc-dap)
DAP Protocol (via http)
18
Our Client Library(prototype)

• One more reason
• Help Clients to view Swath

No Latitude and Longitude
Courtesy of NASA
19
Visualizing HDF-EOS5 Grids
Visualizing HDF-EOS5 Swath Problem
Swath
???
View Swath
View Data (User)
Remote HDF5
Remote HDF-EOS5
HDF5 Handler
HDF5 Handler w/ CF option
Visualization tools (grads)
Visualization Tools (gradsoc)
Server (apache)
Client Library (libnc-dap)
HDF5-Friendly Client Library (liboc-dap)
DAP Protocol (via http)
20
Demo MLS swath via GrADS

• GrADS coupled with our client
• library
• The client library provides grid
• mapping from swath data
• GrADS displays swath through
• HDF-EOS5 specific client library
• API calls

21
Summary of Client Library (Prototype)

• Finished coding
• Tested with the GrADS visualization client
• Documented
• Working on a demo DAP to HDF5 tool by using the
  client library prototype
• Will test with NCL if time allows

22
Caution

• Our Client Library is ONLY A PROTOTYPE!
• It does NOT support all DAP data types
• It does NOT support all AURA files
• It does NOT support all Visualization clients

23
Summary

• HDF5 Access via OPeNDAP
• Is Easy
• Is used by GES DISC to serve Aura
• files
• HDF5-Friendly OPeNDAP Client Library
• Serves HDF5 better(EOS swath)
• Benefits visualization clients

24
Future Work

• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP
• Client Library (Prototype)
• URL http//hdfgroup.org/projects/opendap

25
Credits
Mike Folk (The HDF Group) Robert McGrath
(NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins
(ADNET) Christopher Lynnes, James Johnson, Denis
Nadeau, Fan Fang (NASA) Jennifer Adams
(GrADS) Dave Brown (UCAR)
26
Acknowledgement

• This work was supported basing upon the
  Cooperative Agreement with the National
  Aeronautics and Space Administration (NASA) under
  NASA grant NNX06AC83A, NNX08A077A and
  NNX06AG75A. Any opinions, findings, and
  conclusions or recommendations expressed in this
  material are those of the author(s) and do not
  necessarily reflect the views of NASA.