HDF-EOS%20Status,%20Related%20Tools%20and%20Issues

1
HDF-EOS Status, Related Tools and Issues

• Abe Taaheri, Raytheon IIS
• 2009 THG Briefing to ESDIS

2
TOOLKIT / HDF-EOS Support

• Maintenance support is funded on EMD Evolution
  through May 22, 2009
• - Bridge contract until EODDIS Evolution
  Development (EED) award in 2010
• Current support
• HDF-EOS 2 5
• SDP/MTD Toolkit
• HDF-EOS plug-in for HDFView
• HDF-EOS2 to GeoTIFF Converter (HEG)
• Other Toolkit/HDF-EOS related software
• Toolkit/HDF-EOS/HEG helpdesk

3
ECS Support of HDF-EOS

• Both HDF4 and HDF5 based flavors of HDF-EOS are
  part of ECS baseline.
• OSs Solaris (9, 10), Irix6.5 (64, n32),
• Linux (including 64-bit Opteron and Itanium),
  
• Mac OS X (Power PC and Mac Intel),
• Windows XP(MTD Toolkit, HDF-EOS2 5, HEG,
  HDFView)
• added full Cygwin support for next release
• Compilers Fortran 77/90 g77/pgf90 , C, C,
  gcc, g,
• Intel Fortran (SGI Altix), gfortran (Mac Intel)

4
Downloadable Software

• SDP/MTD Toolkit (Science Data Processing Tools)
• HDF-EOS2 (HDF4 based)
• HDF-EOS5 (HDF5 based)
• heconvert (converts HDF4 - based
  Grid/Point/Swath structures to HDF-EOS 5
  equivalents)

5
Downloadable Applications

• HEG
• - GeoTIFF conversion
• - Spatial subsetting, Field Subsetting
• - reprojection
• - stitching, sub-sampling and more .
• DumpMetadata (Dump HDF-EOS metadata)
• HDFView Plug-in, Java-based browser (HDF4,
  HDF5, HDF-EOS 2 and 5 access)
• HE5View (HDF5 - based browser)
• EOSView (HDF4 - based browser)
• Dropped support for OS updates on July 2006

6
Software Releases

• Toolkit/HDF-EOS
• Last release March 2008
• TOOLKIT 5.2.15
• HDF-EOS 2.15, HDF- EOS5.1.11
• with HDF4.2r3, HDF5-1.6.7, Szip2.1
• Next release May 2009
• TOOLKIT 5.2.16
• HDF-EOS 2.16, HDF-EOS5.1.12
• with new releases of HDF4 (2r4), 5 (1.8.3)

7
Software Releases (Cont.)

• HEG (HDF-EOS2 to GeoTIFF Converter)
• Last release August 2008
• Version 2.9
• Next release May 2009
• Version 2.10
• Already have released Datapool version to the
  DAACs (Feb. 2009)

8
Software Releases (Cont.)

• HDF-EOS Plug-in for HDFView
• Last release September 2008
• Version 2.4
• Next release June 2009
• For HDFView Version 2.5

9
Software Releases (Cont.)

• HECONVERT (HDF-EOS2 to HDF-EOS5 Converter)
• Last releases
• 2001, 2004 Version ?
• Oct. 2008 Version 2.0
• DumpMetadata (ASCII Dump of HDF-EOS metadata)
• Last release
• Dec. 2008 Version 1.0

10
SDP Toolkit

• Science Data Production (SDP) Toolkit is a
  collection of tools used by
• - Science software developers who produce
• code to process instrument data.
• - HDF-EOS users who need to
• Access metadata in HDF-EOS files
• Perform Time/Date conversion
• Access Digital Elevation Model Data
• Access Ancillary Data, such as Digital Chart
• of the World database (DCW), Olson World
  Ecosystem files, DEM data files, etc.

11
SDP Toolkit (cont.)

• Tools provided in (SDP, MTD) Toolkit
• - AA (Ancillary Data Access)
• - CBP (CELESTIAL Body Position)
• - CSC (Coordinate System Conversion)
• - CUC (Constant and Unit Conversions)
• - DEM (Digital Elevation Model Access)
• - EPH (Ephemeris Data Access)
• - GCT (Geo Coordinate Transformation)
• - IO (Input Output (File I/O))
• - MEM (Memory Management)
• - MET (Metadata Access)
• - PC (Process Control)
• - SMF (Status Message File (Error/Status))
• - TD (Time and Date Conversion)

12
What is HDF-EOS ?

• HDF-EOS is an extension to HDF which standardizes
  storage of and access to common Earth Sciences
  data structures
• By providing a single interface to data
  structures common to earth science
• By providing a container for EOS inventory,
  archive and product specific metadata
• By establishing a consistent relationship between
  geolocation and science data.

13
HDF-EOS 2

• HDF4 - based, storage format for EOS standard
  products.
• Used operationally by MODIS, MISR, ASTER,
  Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by THG/NASA
  for the fore-seeable future.

14
HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF-EOS2
  with a different user interface.
• First released in 2000.
• Format used by EOS Aura instruments
• Designed to resemble HDF-EOS 2 to the maximum
  extent possible.
• Supports same data structures
• Added prefix HE5_ to HDF-EOS 2 functions.

15
HDFView

• HDFView A Java-based browser from THG for HDF4
  and HDF5 based data.
• HDF-EOS plug-in extends HDFView to browse HDF-EOS
  data in both versions.
• The extension is a plug-in module and does not
  change THG HDFView functionality. All copy, save,
  editing, image display, etc. for HDF4 and 5
  objects remains.

16
HDFView

• Identical look and feel when displaying HDF and
  HDF-EOS objects.
• Users can access a complete breakdown of HDF-EOS
  objects. e.g. grid information, projection info.,
  dimensions, datafields, and attributes are
  displayed. 
• Current Support SUN, SGI, LINUX, Mac, WINDOWS
  platforms

17
Swath, Grid, and Point Objects
18
AMSR-E Swath
Tableview Features in Version 2.4
19
HDF-EOS to GeoTIFF Converter (HEG)

• HEG Utility that converts EOSDIS data from
• HDF-EOS format to common Geographical
• Information System (GIS)-compatible formats.
  
• Two versions available
• A downloadable desktop version
• Access through NASA archive online storage (Data
  Pools)
• Implemented also in some prototypes
• Stand-alone Data Pool
• OGC (Open Geospatial Consortium) for Web Services
  Chaining
• DOWS (Deploy OGC Web Services) on the DAACs

20
HEG Functionality

• Conversion
• HDF-EOS (Swath or Grid ) to a single-band or
  multi-band GeoTIFFs
• HDF-EOS Swath to HDF-EOS Grid
• HDF-EOS Swath or Grid to generic Binary
• Metadata preservation/creation.
• Allows file selection from users local storage

21
HEG Functionality

• Subsetting (spatial, field, band)
• Stitching (mosaicing) subsetting Reprojecting
• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,
  GEO, SIN, Albers)
• Subsampling (Subsample stacks)
• Running from Command Line (batch jobs)
• or with Java-based GUI

22
HEG Functionality

• Currently supports MODIS, MISR, ASTER, AIRS and
  AMSR-E products on TERRA and AQUA (gt110
  products),
• Operable on Sun, SGI, Win, Linux, MAC
• Integrated into ECS Data Pool
• Reduces the transfer time of HDF-EOS data sets
  (if subset is requested)
• Provides the end-user with the exact file
  required by their application
• User Interface
• Portable, written in Java.
• Not dependent on COTS (eg. IDL).

23
Mt. Etna Eruption MISR L1B Stitched Blocks
24
2004 Hurricane Francis, MODIS
MOD021KM, Multi-Band GeoTIFF, 3 bands
EV_1KM_RefSB, Bands 1,2,5
25
AST_L1B, 2003 California FiresGeo. projection,
Multi-Band GeoTIFF, (3 bands Visible Near
Infrared (VNIR))
.. and Subsetted
Two Swaths Stitched ..
26
Recent Major Development/Fixes

• TOOLKIT/HDF-EOS
• SZIP Insatallation Default decoder,
• Choice for encoder
• Choice for configuring HDF4 with/without SZIP
• Choice for disabling HDF4s internal netCDF
• Choice for installing Toolkit/HDF-EOS in 32 or
  64-bit modes in 64-bit Linux platforms
• (except for Itanium no m32 flag)

27
Recent Major Development/Fixes

• Support for new releases of HDF4/HDF5/SZIP
• (HDF4.2r4, HDF5-1.8.X, SZIP-2.1)
• Auto Configuration capability (like HDF) for
• HDF-EOS2 and HDF-EOS5
• Support for Mac Intel platform.
• Support for CYGWIN

28
Recent Major Development/Fixes

• Added APIs GDij2ll and GDll2ij to HDF-EOS2 and
  HDF-EOS5
• Added HDF_EOS 2 5 utilities for dumping grid
  lat/lon into ASCII or binary files
• Bug fixes such as
• - HDFEOS5 Attribute data type inquiry (code
  returned not useful for reuse)
• - HDFEOS5 Tiling/comp. consistency between
  dataset and structmetadata
• - HDF-EOS2 Correct diminfo for a field (value
  in structmetadata is wrong?)
• - HDF-EOS5 wrong datatype info for INT8 data (C
  code)
• - Toolkit Problem in pccheck utility( hangs if
  a blank space at a line start )
• - Fix for ODL core dump (too many file names for
  Input Pointer)

29
Recent Major Development/Fixes

• HDF-EOS Plug-in for HDFView
• Current Support SUN, LINUX, WINDOWS, and
  MAC (Power and Intel)
• Auto installation of plug-in into HDFView
• running self-extracting archives.
• Adding to plug-in the fixes and enhancement
• that went into HDFView2.4 and 2.5
• Display image for selected pixels in the viewed
  tables

30
Recent Major Development/Fixes

• Display Lat/Lon for a selected pixel in the
• data field tables
• Full Mac OSX support
• Improved Help Window
• New icons for activating Lat/Lon display for
• Swath Grid
• Format conversion HDF-EOS2 -gt HDF-EOS5
• Correction for compression and tiling values for
  grid fields

31
Recent Major Development/Fixes

• HEG
• Product Support
• AMSR-E
• Land products (CEA Projection added)
• AMSR-E 5 day Snow, and Monthly Snow
• Weekly Ocean, and Monthly Ocean
• Daily 12 Km, and Daily 6 Km Sea Ice
• L2 Rain, and Rain Grid
• ASTR_07XT, ASTR_09XT, ASTR_L1BE, MOD29E1D
• More new versions and products tested. Will
  update Users Guide and HEG web site

32
Recent Major Development/Fixes

• Stitching fixes
• MOD14A1 3-D fields (band selection)
• TM grids
• Multi-band geotif/HDF-EOS in Datapool
• Other Fixes
• Interpolation correction with AIRS products
• containing fillvalue in the data fields
• Pixel size unit consistency in conversion and
  stitch GUIs.

33
Recent Major Development/Fixes

• Correction for droping a column when
• stitching MODIS SIN tiles.
• Fixes for processing with default pixel size
  instead of ignoring it for Geographic
  projection
• Spatial subsetting fix for global AMSR-E
  garanules
• Fix for ignoring re-projection of type NN
  interpolation after stitching
• support pixel resampling for gridded data

34
Future Plans

• TOOLKIT/HDF-EOS
• Auto Configuration for TOOLKIT
• XML output for Metadata in TOOLKIT
• Update of DEM land-sea mask
• Support for HDF5-1.8.X
• Validation tool for HDF-EOS5 files

35
Future Plans (cont.)

• HDF-EOS Plug-in for HDFView
• More functionalities
• - Ability to modify input HDF-EOS2 granules.
• - Ability to cut/paste objects, modify/delete
  attributes.
• - Ability to create new grid/swath
• HEG
• - Support for AMSR-E Level 3 Land (CEA
  projection)
• - New MRT (MODIS Reprojection Tool) release
  updates
• - Support HEG for MAC Intel

36
Availability

• Access to libraries, applications, Documentation
  
• TOOLKIT
• http//newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.h
  tml
• HDFView
• http//newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/
• HDFView_hdfeos_plugin.html
• HEG
• http//newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHo
  me.html
• Email
• Abe_Taaheri_at_raytheon.com
• Landover_PGSTLKIT_at_raytheon.com