MOOS SSDS Data Access Features

1
MOOS SSDSData Access Features

• A Discussion with
• MBARIs Science Data Users

2
Todays Goals

• Provide overview of SSDS in MOOS context
• Review user interface (UI) products and concepts
• Show you UI options, and let you guide our way
• Offer opinions on good and bad approaches
• Identify long-term goals and ideas
• Note We are not presenting a released product
• It will be a while (6-12 months) before the
  system is solid
• Just getting early feedback on concepts from
  users
• Part of iterative development process

3
Todays Sequence of Events

• Throwaway Demo (Look, it works, really!)
• Introduce MOOS and SSDS
• Demonstrate Our Concept of Operations
• Show a simulated example using real SSDS
  software. Knowledge increases! Data is stored!
• Demonstrate Prototype User Interfaces
• Tour of (MOOS) HOOVES
• You Tell Us What Are the Most Wanted MOOS Data
  Products
• Fill out the survey, join ssdsusers, just talk to
  us
• Well keep improving (read changing) these
  products for a while, but well try to give fair
  warning.

4
Demonstrating Real Data

• Interface to MOOS Test Mooring data
• Put together quickly, using assumptions
• This interface is simple because data is still
  simple
• Data was collected through latest MTM run
• As data sets get more complicated, this kind of
  interface breaks down
• Subsetting, varied record types, deployment
  changes all make the data rather messy
• At end of presentation you can tell us what you
  need to work in the real world

5
MOOS (Shows Data Flow)
Applications/ Interfaces
User
Communications
            Portal
Deployed Platform
Data Tracking
Applications
Devices
Shore Side Data System

(User Tools)
Ocean Side
Shore Side
6
Shore Side Data SystemRequirements

• Ingest data in any described format and save it
• Capture, publish data descriptions (via metadata)
  
• Provide standards-based access to data
• Raw data, and other common digital formats
• APIs for use by common visualization and analysis
  tools
• User-oriented web interfaces, with quick-look
  plots
• Merge data from different sources time
  intervals
• Allow straightforward visualization of data
• Simplify data processing and product archiving
• Provide for automated generic quality checks
• Provide data access security as needed

7
Johns Top 10 List Why Use SSDS?

• 10. All MOOS packets come via ISI. Well format
  and put the raw data back together for you.
• 9. All the annoying stuffbackups, permissions,
  storage, namingis handled by us (for free!).
  
• 8. You can point other people to your data.
• 7. You can see other peoples data.
• 6. You can actually search for data (by name!).
• 5. You can easily get basic () plots of your
  data.
• 4. You wont ever have to remember whats in your
  data again, it will all be described. Forever.
• 3. It will plug data in to your favorite
  application.
• 2. Every data access tool we build works for you.
• 1. Everybody else will be doing it.

8
How Does It Work?

• First, you describe whats in your data records.
  (This is the metadatamore about it later.
• That description is stored with (near) your
  instrument, and sent to SSDS before any data from
  your instrument. SSDS tracks this info.
• When you send a data record, identify its type.
• SSDS automatically routes data of each type to
  its own data bucket.
• SSDS automagically knows all about your data
  records, because you described them. Now it can
  do all sorts of useful things with your data
• Plot ? Print ? Search ? Merge
• Format (on request) ? Describe (in files
  headers)
• Send to Applications ? Point to by variable
  name

9
Metadata Description Example

• ltrecordDescriptiongt
• ltrecordTypegt1lt/recordTypegt
• ltbufferStylegtasciilt/bufferStylegt
• ltbufferLengthTypegtvariablelt/bufferLengthTypegt
• ltbufferParseTypegtseparatedlt/bufferParseTypegt
• ltbufferItemSeparatorgt,lt/bufferItemSeparatorgt
• lt!-- crlf terminates a record --gt
• ltrecordTerminatorgt\r\nlt/recordTerminatorgt
• ltvariablegt
• ltcolumnIndexgt1lt/columnIndexgt
• ltformatgt3slt/formatgt
• ltvariableNamegtmessageTypelt/variableNamegt
• ltunitsgtlt/unitsgt
• ltlong_namegtMessage Typelt/long_namegt
• ltdescriptiongtWB indicates house keeping
  messagelt/descriptiongt
• lt/variablegt
• ltvariablegt
• ltcolumnIndexgt2lt/columnIndexgt
• ltformatgtilt/formatgt

10
Metadata Description Example

• ltvariablegt
• ltcolumnIndexgt3lt/columnIndexgt
• ltformatgt6slt/formatgt
• ltvariableNamegtdatelt/variableNamegt
• ltunitsgtYYMMDDlt/unitsgt
• ltlong_namegtDate in YYMMDD formatlt/long_namegt
• lt/variablegt
• ltvariablegt
• ltcolumnIndexgt4lt/columnIndexgt
• ltformatgt4slt/formatgt
• ltvariableNamegttimelt/variableNamegt
• ltunitsgtHHMMlt/unitsgt
• ltlong_namegtTime in HHMM formatlt/long_namegt
• lt/variablegt
• ltvariablegt
• ltcolumnIndexgt5lt/columnIndexgt

11
Data Record Identification

• Each data record sent via ISI is wrapped with
  record header metadata to identify it
• Contents of record header metadata include
• Data source ID (all MOOS data sources have ID)
• Metadata ID (what file has a description to this
  record)
• Record Type ID (which kind of record is this)
• Timestamp (provided by ISI when record is
  submitted)
• System now knows what to do with each data
  record (otherwise theyd all be mushed together
  in a long stream of records)
• Right now we also ask for and use source
  platform
• Exact mechanism used to supply this is TBD

12
Recap MOOS Data Flow
Applications/ Interfaces
User
Communications
            Portal
Deployed Platform
2c
Data Tracking
Applications
4
3
2b
5a
2a
1
5b
Devices
Shore Side Data System

(User Tools)
Ocean Side
Shore Side
13
SSDS Elements
Automated Data Flow
Internal Interfaces
On-Demand Interactions
(Re)Processed and New Data Sets
Arriving Data
Metadata
Data Tracker
Ingest
lt Requests Data gt
110 234 999 223 207 191
Archiving
Web I/Flt Requests Data gt
Data Presentation
Data line 1 more data last data
Data For Analysis
Data Catalog
External Data Stores
Shared Descriptions
14
Standard Interfaces
(and Tools)
(Re)Processed and New Data Sets
FGDC
Arriving Data
Metadata
Data Tracker
Ingest
lt Requests Data gt
XML/ DTDs
110 234 999 223 207 191
SQL
Archiving
Web I/Flt Requests Data gt
Data Presentation
DODS
Ferret
LAS
netCDF
Data line 1 more data last data
inGrid
HTTP
Browsers
ncBrowse
Data For Analysis
Z39.50
Data Catalog
DODS
External Data Stores
Shared Descriptions
15
Science Data UsersWhat Do You Need (First)?

• Raw data via device ID pages? (sort of limited)
• Standard plots the same as OASIS quality control?
• Standard plots with multiple plots per page?
• Access data from applications via a DODS URLs?
• Matlab, Ingrid, Live Access Server, Excel, IDV,
  Ferret
• And hopefully, Ocean Data View
• Access data via shipped data files (e.g., ASCII
  CSV w/headers) opened within desktop
  applications?
• Excel, ArcView, Ocean Data View
• Delivery of data directly into an application?
• Ability to subset data, for example by time
  window?
• Ability to merge data from different data sets?

16
Summary

• Good data descriptions of structured data records
  will make many things possible
• Many capabilities will be possibleneed your
  input to determine priorities
• This project is iterative, so we can fix,
  improve, and add high priority features
• Well present some examples as food for thought,
  then open it up for discussion

17
References

• What Data is Available?
• Metadata Explained
• How To Access the Data?
• How to Display the Data?
• MOOS Data Architecture
• How to Access Instrument
• How Data Access Works

• How Easy to Use is It?
• Are we sure the data is OK?
• What if theres a problem?
• Is data distributable?
• Is data secure?
• What arent you getting?

18
Example of System In Action

• Developer Get Device ID
• Developer Define instrument using XML
• Instrument Start up, send metadata into system
• SSDS Receive metadata, update instrument info
• Instrument Send data record(s) into system
• SSDS Receive data, save, make available to
  users
• User/Operator Access data