Unidata: Data, tools, and community leadership for enhanced Earthsystem education and research

1
Unidata Data, tools, and community leadership
for enhanced Earth-system education and
research
Ben DomenicoJune 2003

• Operated by the University Corporation for
  Atmospheric Research In Boulder, Colorado
  http//www.ucar.edu
• Sponsored by the National Science Foundationof
  the U.S. Government http//www.nsf.gov

2
Types of Data

• Traditional observations from weather stations
  around the globe
• Satellite imagery
• Radar data from 150 NEXRAD radars
• Output from forecast model runs at the National
  Centers for Environmental Prediction
• Lightning strike data
• Measurements from sensors on commercial aircraft

3
1Km Radar Image
4
Unidata Community Today

• More than160 institutions
• Includes over 100 academic institutions plus
  government agencies and private sector research
  groups
• Does not count separate installations, e.g.
  Spanish weather service implementation of IDD
• Interdisciplinary from the outset 1996 survey
  showed over 2/3 of institutions had some uses
  outside meteorology (oceanography, hydrology,
  climatology, civil engineering, environmental
  science)

5
Impact Survey

• Over 21,000 college students per year use Unidata
  tools and data in classrooms and labs
• Nearly 4,000 women/minority students
• More than 1,800 faculty and research staff
• Over 55,000 K-12 students involved through
  Unidata-connected university programs
• Informal education in excess of 1 million hits
  at Unidata-based university web sites per day
• 97 of community report being satisfied or very
  satisfied

6
Principal Activities of the Unidata Program
Center

• Facilitating Data Access to a broad spectrum of
  observations forecasts (in near real time)
• Providing Tools to visualize, analyze, organize,
  receive, share data at university sites
• Supporting Faculty who use Unidata systems at
  colleges universities (most in the U.S.)
• Building and Advocating for a Community where
  data, tools, best practices in
  education/research are shared

7
IDD The Community in Action

• The Internet-based system by which universities
  acquire huge quantities of weather data in
  near-real time (i.e. ASAP) typifies Unidatas
  community orientation.
• The system has no data center -- all tasks are
  performed on the participants own (small)
  computers.
• Currently the most used advanced application on
  the Abilene network (2-3 in terms of packets and
  bytes transferred)

8
Internet Data Distribution (IDD)with Multiple
Sources (Injecting 17 Gigabytes per Day)

• Using LDM software for instant data relaying,
  160 institutions cooperate to acquire a wide
  range of real-time, global, atmospheric oceanic
  observations, model outputs, remotely sensed
  images..., in a coordinated community effort.

9
IDD Fanout from Source for Scalability
10
Real-time Data for Local Weather, Flood,
Dispersion, Models
Satellite imagery
Weather station observations
High ResolutionLocal Model
Radar data assimilation
Demographic data for decision makers
Stream gauge measurements
11
Typical Data Handlingat a Unidata Site
Unidata user running McIDAS, GEMPAK, MetApps
Unidata user running McIDAS, GEMPAK, MetApps
Weather station observations
Application specific protocols
Decoders
Satellite imagery
Decoders
IDD
Local data decoded into application specific
formats
Radar data
Decoders
Decoders
Lightning, aircraft, GPSmet, etc.
Decoders
12
THREDDSTHematic Realtime Environmental Data
Distributed Services

• NSDL Collection
• Funding via NSF Division of Undergraduate
  Education
• Integrates environmental data and tools into
  digital libraries (metadata technology)
• Infrastructure for embedding pointers to data and
  interactive analysis tools into online
  educational materials and scientific publications
• 20 major partners data providers, visualization
  tool builders, metadata experts.

13
Thematic Data Servers (combining IDD push with
several forms of pull and DL discovery)
User applications e.g., LAS, McIDAS,
MetApps/VisAD, IDL, MatLab...
DLESE Digital Library for Earth-System Education
Discovery
DODS, ADDE, FTP protocols
DLinterchangeprotocol
Satellite Images, e.g.
Satellite Images, e.g.
Satellite Images, e.g.
Hydrology Data, e.g.
Geophysical Data, e.g.
Satellite Imagery...
IDD
IDD
IDD
IDD
IDD
14
Integrated Visualization of Data from Thematic
Servers
SUNYIDD
Lightningdata server
SSECIDD
Satellitedata server
Streamflowdata server
Client/server data access
NOAAport IDD
Conventionaldata server
USGS?IDD
Demographic data server
Radardata server
Global modeldata server
Local model computation anddata server
CRAFTIDD
CONDUITIDD
15
(No Transcript)
16
People
Remote Catalog Query and Data Access via Local
Analysis Tool
Analysis and Visualization Tools
Discovery andPublication Tools
User accesses remote catalog via local analysis
tool
User accesses remote dataset via local analysis
tool
Discovery and Publication Services
Data Services
Data Services
CatalogGeneration Tools
Documents
Data
17
Analysis and Display Software
Compound Documents
i.e.,electronic publications with embedded
pointers to datasets and tools
Discovery CenterCatalog Systemse.g. DLESE,
NSDL, GCMD
e.g. IDV, VGEE,LAS, INGRID, IDL, GDS
Discovery and Usage Metadata Middleware e.g.
THREDDS catalog generators, servers, harvesters
Data Server Software e.g. FTP, OPeNDAP, ESG,
OpenGIS, Web services
Data Storage System e.g., Disk, RAID, Mass
storage)
18
Browser-base Thin Client Access

• LDEO/IRI web site publishes catalog of datasets
  available on server at UCAR
• Catalog resides and is updated at UCAR
• Browsing of datasets on UCAR server from LDEO
  server
• Also enables analysis and display of datasets on
  UCAR server using tools on LDEO server

19
THREDDS, GIS, DL Interoperability
THREDDS Client Applications
GIS Client Applications
OpenGIS Protocols WMS, WFS, WCS
OGC or OPeNDAPADDE. FTP protocols
OGC or proprietary GISprotocols
Metadatacrosswalk
Metadatacrosswalk
Open Archives Initiative (OAI) Metadata Harvesting
Digital Library Discovery Systems
20
Enhanced Catalogs
Digital Library Catalog System
ESML or NCML Generator
IDD
Data Catalog Harvesting
Enhanced Catalogs
Inventory Catalog Generator
Enhanced Inventory Catalog and Data Server
THREDDS Inventory Catalog and Data Server
THREDDSData Server
IDD
Data Mining Engine
IDD
Case Study Catalog
Third Party THREDDS Catalog Server
Event Gazetteer
Third Party THREDDS Catalog Server
Third Party THREDDS Catalog Server
21
Assimilation
IDD
Local weather forecast model
Initialization
Assimilation
Coupled hydro model
Assimilation
Local weather forecast model
22
(No Transcript)
23
Interesting Weather
24
Partnerships via THREDDS

• NCDC and NOMADS at National Climatic Data Center
• NGDC, National Geophysical Data Center
• SSEC, Space Science and Engineering Center at
  University of Wisconsin
• IRI/LDEO, International Research Institute/Lamont
  Doherty Earth Observatory
• PMEL, the Pacific Marine Environment Laboratory
• CDC, the Climate Diagnostic Center
• FNMOC, Fleet Numerical Meteorological and
  Oceanographic Center
• GMU/COLA, George Mason University/Center for
  Oceans Land Atmosphere
• University of Alabama Huntsville and NASA
  Marshall
• IRIS DMC, Incorporated Research Institutes for
  Seismology Data Management Center
• University of Oklahoma
• ARM, Atmospheric Radiation Measurement
• University of Florence
• GCMD, Global Change Master Directory at NASA GSFC
• JPL, Jet Propulsion Laboratory

25
THREDDS 2G Additions

• Data providers
• CIESIN (Consortium for International Earth
  Science Information Network, Bob Downs)
• CUAHSI (Consortium of Universities for
  Advancement of Hydrologic Science, David
  Maidment)
• ESIG/NCAR (NCAR Environmental Societal Impacts
  Group, Bob Harriss)
• Earthscope (UCAR UNAVCO, Chuck Meertens)
• GEON (GEOphysical Network, Chaitan Baru, UCSD San
  Diego Supercomputer Center)
• ESRI GIS Community
• Analysis and Display Tools
• ESRI GIS Clients (ESRI, Inc., Jack Dangermond,
  President)
• OGC Clients (Open GIS Consortium, David Schell,
  President)
• MyWorld (Northwestern educational GIS Client,
  Danny Edelson)

26
THREDDS 2G GIS and Web Services Interoperability

• ADL (Gazetteer Services The University of
  California, Santa Barbara, Linda Hill and Michael
  Goodchild)
• DLESE Evaluation Services (The University of
  Colorado CIRES, Susan Buhr)
• DLESE Data Services (Tamara Ledley)
• DLESE Program Center Digital Library for Earth
  System Education (Mary Marlino)
• ESRI (Jack Dangermond, President)
• OPeNDAP (The University of Rhode Island Open
  source Project for a Network Data Access Protocol
  -- formerly DODS, Peter Cornillon)
• LAITS (Laboratory for Advanced Information
  Technology and Standards,Liping Di, George Mason
  University)
• NSDL Evaluation Services (University of Colorado,
  Tamara Sumner)
• DLESE Evaluation (Colorado, Georgia, Susan Buhr)
• OGC (Open GIS Consortium, David Schell,
  President)
• SWEET (Semantic Web for Earth and Environmental
  Terminology, Rob Raskin)

27
Enhanced (via ESG) Metadata Catalog
28
ISCCP Collection Metadata
29
More Information

• http//my.unidata.ucar.edu/
• http//www.unidata.ucar.edu/projects/THREDDS/
• ben_at_unidata.ucar.edu