OPeNDAP The Opensource Project for a Network Data Access Protocol BOM Tutorial Introduction October

1
OPeNDAP(The Open-source Project for a Network
Data Access Protocol) BOM Tutorial
IntroductionOctober 15, 2007pfox_at_opendap.org
2
Distributed Oceanographic Data System (DODS)

• Conceived in 1993 at a workshop held at URI.

• Objectives were
• to facilitate access to PI held data as well as
  data held in national archives and
• to allow the data user to analyze data using the
  application package with which he or she is the
  most familiar.

• Basic system designed and implemented in
  1993-1995 by Gallagher and Flierl with NASA and
  NOAA funding.
• From 1994 to present it has been extended with
  NASA, NOPP, NSF and NOAA funding.

3
Some Definitions

• DAP Data Access Protocol
• Model used to describe the data
• Request syntax and semantics and
• Response syntax and semantics.
• OPeNDAP
• The software
• Numerous reference implementations
• Core/libraries and services (servers and
  clients).
• OPeNDAP Inc.
• OPeNDAP is a 501.c(3) non-profit corporation
• Formed to maintain, evolve and promote the
  discipline neutral DAP that was the DODS core
  infrastructure.

4
Some Definitions

• Syntax
• The computer representation of a data object -
  the data types and structures at the computer
  level e.g.,
• T is a floating point array of 20 by 40 elements.
  
• Semantics
• The information about the contents of an object
  e.g.,
• T is sea surface temperature in degrees Celsius
  for a certain region of the Earth.

5
Considerations with regard to the development of
DAP and OPeNDAP

• Many data providers

• Many data formats

• Many different client types

• Many different semantic representations of the
  data
• Many different security requirements

6
Broad Vision
A world in which a single data access protocol is
used for the exchange of data between network
based applications regardless of discipline. A
layer above TCP/IP providing for syntactic and
semantic consistency not available in existing
protocols such as FTP.
7
Practical Considerations

• The broad vision
• Is syntactically achievable, but
• Is not semantically achievable, at least not in
  the near term.

8
OPeNDAP Inc. Mission Statement
To maintain, evolve and promote a data access
protocol (DAP) and reference implementation
software (OPeNDAP) for the syntactically
consistent exchange of data over the
network. The DAP should provide syntactic
interoperability across disciplines and allow for
semantic interoperability within disciplines.
9
OPeNDAP Inc. Vision Statement

• To achieve the mission
• OPeNDAP Inc. will be non-profit.
• Easier to obtain federal funds.
• The DAP is more likely to be adopted.
• OPeNDAP software will be open source.
• More likely to be adopted.
• Need community contributions to software.
• OPeNDAP Inc. will mix implementation with
  research.
• Implementation - to encourage use.
• Research - to keep the protocol current.

10
OPeNDAP Inc. Vision Statement (cont)

• OPeNDAP Inc. will rely primarily on federal
  funding.
• Unlikely to obtain private funding for
  middleware.
• Development to be use case driven.
• Aligned with Vision/Mission.
• Strategic direction will be sought from an
  Advisory Board consisting of data system experts
  and with input from you the community of
  developers.
• OPeNDAP Inc. will seek partners.
• OPeNDAP Inc. will utilize community working
  groups to develop standards related to the DAP,
  OPeNDAP

11
The Data Access Protocol (DAP)

• The DAP has been designed to be as general as
  possible without being constrained to a
  particular discipline or world view.

• The DAP is a discipline neutral data access
  protocol it is being used in astronomy,
  medicine, earth science,
• Provides data format and location, and data
  organization transparency
• Is metadata neutral

12
DAP comparisons

• File-based
• GridFTP/FTP
• HTTP
• SRB
• Service-based
• Open-Geospatial Consortium, WCS, WMS, WFS,
• Virtual Observatory (Astronomy), SIAP, SSAP,
  STAP,

13
The Astronomy approach data-types as a service

• VOTable
• Simple Image Access Protocol
• Simple Spectrum Access Protocol
• Simple Time Access Protocol

VO App2
VO App3
VO App1
OGC WFS, WCS, WMS and SWE SOS, SPS, SAS use
the same approach
VO layer
DBn
DB2
DB3

DB1
14
Who is using DAP/ OPeNDAP?

• Science examples
• PMEL with their Tsunami inundation modeling
• Ocean regional modelers to extract open boundary
  conditions
• Visualization of data sets using MATLAB/IDL/
• Service examples
• Live Access Server
• Mapserver OGC services and OPeNDAP data access
  (future)
• Digital Library Service - metadata and catalogue
  info

15
Data Access Protocol (DAP2) - Current

• DAP2 currently a NASA/ESE Standard
• Current servers implement DAP2

DAP3

• DAP 2 XML responses (implemented)

16
DAP4 - Late 2007 (?)

• DAP4 improvements over DAP3
• Additional datatypes
• Swath
• Blob - GIF, MPEG,
• Additional functionality
• Check sum
• Modulo
• The additional datatypes will enable the DAP to
  be used in a wider variety of circumstances and
  are a direct response to users requests.

17
What DAP means to me

• Data access and transport
• Response types DAP objects versus file type
• A DAP URL is essentially an HTTP URL with
  additional restrictions placed on the abs-path
  component.
• DAP2-URL "http//" host "" port
  abs-path
• abs-path server-path data-source-id "." ext
  "?" query
• server-path "/" token
• data-source-id "/" token
• ext "das" "dds" "dods"
• The server-path is the pathname to the server,
  whereas data-source-id is the pathname to the
  data.

18
OPeNDAP V3 Architecture
Cgi style access
Data
Client

• CGI-style access
• Uses web server
• HTTP protocol
• Several request and response types
• Reads data files, Databases, et c., returns info
• May return DAP2 objects or other data
• Client can be application, web browser or
  specialized server/service

19
OPeNDAP V4 (Hyrax) Architecture
OLFS
BES
Data
Client

• OPeNDAP Lightweight Front end Server (OLFS)
• Receives requests and asks the BES to fill them
• Uses Java Servlets
• Does not directly touch data
• Multi-protocol

• Back End Server (BES)
• Reads data files, Databases, et c., returns info
• May return DAP2 objects or other data
• Does not require web server

20
Binaries Generated

• There are approximately 80 binaries built on a
  nightly basis. They are built for the following
  platforms/operating systems
• Linux
• FC4
• FC5
• MacOS-X (universal binaries when possible)
• Windows XP, win32
• Java 1.5 (Tomcat 5.5)
• IRIX (in four variants), Solaris, AIX, OSF

21
Communication

• Website (http//www.opendap.org and
  test.opendap.org)
• SVN - Code repository (http//scm.opendap.org8090
  /trac/browser/)
• Trac - Task/milestone repository
  (http//scm.opendap.org8090/trac/)
• Telecons
• Management - Weekly
• Developers - Weekly on Monday 11am MT
• Twiki -gt MediaWiki, for
• Management
• Developers
• Coming soon http//docs.opendap.org
  (documentation on-line in MediaWiki form)
• opendap-tech_at_opendap.org e-mail list
• opendap_at_opendap.org for announcements (low vol.)

22
Releases/Support

• Periodic releases (announced on www.opendap.org
  and on email lists)
• Support
• OPeNDAP software (support_at_opendap.org)
• opendap-tech_at_opendap.org discussions

23
OPeNDAP and security

• Both administrators and developers must consider
  security aspects of OPeNDAP software
• First developer telecon each month addresses
  security
• This will be addressed during this week

24
OPeNDAP and authentication

• ESG II/III
• Australian Access Federation (AAF)
• .htpasswd
• Tomcat authentication

25
ESG Security Services
26
ESG Security Services - Client Interaction
27
OPeNDAP Community Working Groups

• Modeled after best practices of IETF, W3, OGC,
  IEEE, ISO, and others
• Working Groups
• Authentication
• Security
• Server-side Functions
• Virtualization (Aggregation)
• Server-side processing
• Geospatial Interoperability
• Hyrax and DS (TDS, GDS, FDS, etc.)
• Semantics
• DAPPER
• netCDF C client
• Response types
• Metrics
• Asynchronous transactions
• DAP4
• Relational Database access via DAP
• Wiki (http//docs.opendap.org/index.php/Working_Gr
  oups)

28
OPeNDAP Community Working Groups

• Terms of Reference http//docs.opendap.org/index.p
  hp/Terms_of_Reference
• Abridged version
• Each OPeNDAP Working Group is established to
  apply members' expertise in their focus area to
  produce specific deliverables or outcomes
• Types Software and Documentation
• Minimum 3 people
• 6 month time-frame

29
OPeNDAP System Elements
The OPeNDAP data access protocol is used by a
variety of system elements.

• Clients
• Browser Interfaces
• Data System Integrators (ODC)
• Servers
• Processing Servers
• Aggregating Servers - OPeNDAP chains
• Ancillary Information Services

30
Clients

• Clients make requests and receive responses via
  the DAP.

• Clients convert data from the OPeNDAP data model
  to the form required in the client application.

31
OPeNDAP Clients
Internet
NCL Client
Matlab Client
Web Browser
OPeNDAP Data Connector
Matlab
NCL
32
OCAPI

• A pure OPeNDAP C API (OCAPI) for the client-side
• Applications
• DAP-aware commands for commercial analysis
  programs (e.g., IDL)
• Scripting tools (e.g., Perl)

33
OPeNDAP System Elements
The OPeNDAP data access protocol is used by a
variety of system elements.

• Clients
• Browser Interfaces
• Data System Integrators (ODC)
• Servers
• Processing Servers
• Aggregating Servers - OPeNDAP chains
• Ancillary Information Services

34
Browser interfaces
35
OPenDAP Data Connector
- a Data System Integrator
36
OPeNDAP System Elements
The OPeNDAP data access protocol is used by a
variety of system elements.

• Clients
• Browser Interfaces
• Data System Integrators (ODC)
• Servers
• Processing Servers
• Aggregating Servers - OPeNDAP chains
• Ancillary Information Services

37
Servers

• Servers receive requests and provide responses
  via the DAP.

• Servers convert the data from the form in which
  they are stored to the DAP.

• Servers provide for subsetting of the data and
  more.

38
OPeNDAP Servers
CDM
Flat Binary
HDF4
HDF5
SQL
FITS
CDF
CEDAR
ESML
Data
Data
Data
Data
Data
Data
Data
HDF5
FreeForm
FITS
HDF4
JDBC
CDF
CEDAR
Internet
39
OPeNDAP Servers (specialized processing)
Internet
40
Servers

• Servers may also provide other services
• Directory traversal.

• Browser-based form to build URL.

• Ascii or other representations of data.

• Metadata associated with the data.

• Server side functions.

41
OPeNDAP Aggregation Servers
Internet
42
The Aggregation Server An Example
netCDF Data Set
DSP Data Set
File
File
File
File
File
File
DSP
Aggregation Server
Matlab Client
Matlab
43
OPeNDAPs Hyrax (Server4)

• Uses a modular architecture to support different
  application-level protocols
• Data access using DAP2 (DAP3)
• Catalogs using THREDDS
• Browsing using HTML and ASCII
• Modules for data access
• Different file types
• Potential for database and scripting
• Modules for commands
• Commands provide varying operations for different
  protocols

44
OPeNDAP V4 (Hyrax) Architecture
OLFS
BES
Data
Client

• OPeNDAP Lightweight Front end Server (OLFS)
• Receives requests and asks the BES to fill them
• Uses Java Servlets
• Does not directly touch data
• Multi-protocol

• Back End Server (BES)
• Reads data files, Databases, et c., returns info
• May return DAP2 objects or other data
• Does not require web server

45
OPeNDAP Lightweight Front end Server
GridFTP DAP2
Request Formulation
Request from client
HTTP DAP2
DAP2 (GridFTP, HTTP)
BES
SOAP-DAP (HTTP)
THREDDS
Response to client
46
BES
BES Framework
Network Protocol and Process start/stop activities
PPT
Initialization/ Termination
DAP2 Access
Data Catalogs
BES Commands/ XML Documents
Commands
NetCDF3
HDF4
FreeForm
Data Store Interfaces

PPT is built in (other protocols) Some
commands are built in
Data
Data
Data
47
Additional material
48
OPeNDAP relies on projects

• To guide user-based requirements for application
  needs as well as OPeNDAP core development (use
  cases)
• To provide maintenance and evolution of the core
  software and documentation
• Currently 7 active projects, covers next 2-3
  years, 2 pending proposals

49
Success

• What constitutes success of the OPeNDAP mission?
• Adoption of the DAP across a broad range of
  disciplines with extensive use in several of
  these.
• In order to achieve this the DAP must do the
  following
• It must be sufficiently flexible, all
  encompassing, etc. that it can be used across a
  broad range of data types.
• Its implementation must be robust, secure, easy
  to use, provide for a broad range of services,
  etc.
• The funding stream must be robust.
• Active and engaged developer and user community

50
Risks

• It is still the case that
• Some other data access protocols are seen as more
  attractive regardless of whether or not they are,
  or
• Other data access protocols are developed because
  their community is not aware of OPeNDAP or of
  what its capabilities are.
• We will compare and contrast some of these today

51
To Succeed

• OPeNDAP must make sure that
• Its (server and client) capabilities are well
  known across a broad range of disciplines.
• The data model is inclusive.
• The implementation is robust and meets users
  needs.
• The DAP coexists with other protocols.
• It has a robust funding base.
• It has extensive documentation.

52
Robust Funding?

• What is an appropriate level of activity
  (funding) for OPeNDAP? What should OPeNDAP be
  doing?
• Core only? Clients and servers? Demonstration
  projects?
• Should OPeNDAP be a small staff (core only) or?
• What priority should be assigned to the elements
  currently being developed? Which of these go
  beyond the core?
• DAP4
• AIS - metadata consistency
• THREDDS - Catalogs

• Toolkits - Matlab, IDL,
• Grid capabilities
• OTHERS?

53
Ancillary Information Service

• Current capability Attributes only
• Client-side only
• Local and remote resources
• Local resource databases
• The AIS enables users to augment the metadata for
  a data source in a controlled way without
  requiring write access to the original data. By
  using the DAP, users are also isolated from data
  format issues.

54
AIS enhancements

• Remote resource databases
• AIS server
• AIS for variables
• These enhancements will greatly expand the
  usefulness of the AIS Remote resource databases
  and an AIS server will enable third-party AIS
  sites which may be sponsored by project offices
  or institutions, et cetera.
• AIS for variables will enable adding metadata
  which are stored as data.

55
Proposed AIS Server
0
1
Client linked w/DAP Software
Data Source
AIS Server
3
2
AIS Resource

• 0. Client requests metadata from the AIS server
  (which appears no different from any other DAP
  server).
• 1. The AIS server gets metadata from data source
• 2. The AIS server gets matching the AIS resource
  using the AIS database and merges it into the
  metadata.
• 3. The AIS server returns resulting the metadata
  object.