GADS: A Web Service for accessing large environmental data sets

1
GADS A Web Service for accessing large
environmental data sets

• Jon Blower, Keith Haines, Adit Santokhee
• Reading e-Science Centre
• University of Reading
• Andrew Woolf, RAL

http//www.resc.rdg.ac.uk resc_at_rdg.ac.uk
2
Background

• At Reading we hold copies of various datasets
  (2TB)
• Mainly from models of oceans and atmosphere
• Also some observational data (e.g. satellite
  data)
• From Met Office, SOC, ECMWF, more
• We serve these datasets to many end users
• Scientists (1000s of hits per year)
• Industry (e.g. British Maritime Technology)
• Datasets are in a variety of formats
• netCDF, GRIB, HDF, HDF5
• Data do not conform to naming conventions
• E.g. temp instead of sea_water_potential_temper
  ature

3
Background (2)

• There is a clear need to make access to these
  datasets easier
• Users shouldnt have to know details of how data
  are stored
• Hence development of GADS (Grid Access Data
  Service)
• Developed as part of GODIVA project
• Grid for Ocean Diagnostics, Interactive
  Visualisation and Analysis
• NERC e-Science pilot project
• Originally developed by Woolf et al (2003)
• Allows richer queries and more flexibility than
  DODS standard
• Although we plan to implement a DODS translation
  layer

4
GODIVA Web Portal

• Allows users to interactively select data for
  download using a GUI
• Users can create movies on the fly
• cf. Live Access Server

5
Advantages of GADS

• Users dont need to know anything about storage
  details
• Can expose data with conventional names without
  changing data files
• Users can choose their preferred data format,
  irrespective of how data are stored
• Behaves as aggregation server
• Delivers single file, even if original data
  spanned several files
• Deployed as a Web Service
• Can be called from any platform/language
• Can be called programmatically (easily
  incorporated into larger systems), workflows
• Java / Apache Axis / Tomcat

6
Architecture
7
Metadata structure
8
GADS Methods

• dataQuery() is used for querying the data
  holdings
• What datasets are there?
• What variables are there in the dataset X?
• dataRequest() is used for downloading data
• User can choose the data format
• Can easily download subsets of data
• Uses start-stride-count semantics (familiar in
  community)
• dataRequestNatural()
• Same as dataRequest() but in natural units
  (degrees, metres )

9
dataQuery examples of use

• dataQuery(dataset, variable, axis) general form
• dataQuery(, , ) gets all dataset names in
  the catalogue
• dataQuery(FOAM_NINTH, , ) gets all the
  variable names in the FOAM_NINTH dataset
• dataQuery(FOAM_NINTH, temperature, ) gets
  the details of the grid for the temperature
  variable
• dataQuery(FOAM_NINTH, temperature, z)
  gets all values that the z coordinate can take
• dataQuery(, temperature, ) gets all
  datasets that contain the temperature variable

10
dataRequest example of use

• dataRequest(FOAM_NINTH, temperature, CDF,
• t, 0, 1, 20,
• z, 0, 1, -1,
• y, 100, 4, 400,
• x, 300, 4, 600)
• dataRequestNatural(FOAM_NINTH, temp, CDF,
  t, 2004-06-01 000000, 2004-06-22
  000000, z, 0, 10,
  
  y, 42, 64,
  x, -26,
  9)
• Returns URL to extracted dataset

11
Metadata manager (in progress)
e.g. Adding a dataset can harvest metadata
from netCDF file headers
12
Limitations

• Assumes one timestep per file
• Hence doesnt handle timeseries well
• Long queries can cause problems (synchronous)
• Needs a queuing system
• Rotated grids a problem (esp. for
  dataRequestNatural())
• Could have richer metadata queries

13
Application Search and Rescue

• Search And Rescue Information System (SARIS)
• British Maritime Technology (BMT)
• Used by Coastguard to locate people who have
  fallen overboard
• Runs a model using wind and surface current data
• Forecasts where person will be by the time rescue
  arrives
• By incorporating GADS, SARIS consumes up-to-date
  Met Office forecasts on demand.
• Should improve quality of prediction
• Still in development as a prototype