Title: Comprehensive Large Arraydata Stewardship System CLASS http:www.class.noaa.gov
1Comprehensive Large Array-data Stewardship System
(CLASS) http//www.class.noaa.gov
- What is CLASS and How Will It Serve NOAA
Clientele? - 12 May 2005
- Richard G. Reynolds / CLASS Project Manager
2Agenda
- Data Centers and Data Stewardship
- CLASS System Introduction
- Project Management
- Accomplishments and Near-term Goals
- Budgets and Campaigns
- Functionality User Services
3NOAAs National Data Centers
- NOAAs National Data Centers are major archive,
access, and assessment sites maintaining,
processing, and distributing environmental and
geospatial data. - National Climatic Data Center
WWW.NCDC.NOAA.GOV - Asheville, NC
- National Coastal Data Development Center
WWW.NCDDC.NOAA.GOV - Stennis, MS
- National Geophysical Data Center
WWW.NGDC.NOAA.GOV - Boulder, CO
- National Oceanographic Data Center
WWW.NODC.NOAA.GOV - Silver Spring, MD
4NOAAs National Data Centers(Continued)
- These Centers provide long-term stewardship for
most of NOAAs environmental and geospatial data,
and a broad range of user services. - Centers of data are facilities where extensive
collections of given environmental parameter(s)
are maintained because of individual or
institutional research or operational
requirements - They also serve as Agency Record Centers
5NOAAs National Data Centersare Environmental
Data Stewards
Scientific Data Stewardship is ownership,
knowledge, utilization, and application of the
data CLASS is the Information Technology
infrastructure (hardware and software
environment, and tools) underpinning SDS Data
Rescue preserves and makes available historical
data sets from obsolete media
6Scientific Data Stewardship (SDS)
- Observing system performance monitoring
- Bias monitoring
- Data character and metadata
- Climate Data Records (CDRs) transition from
research to products and services - Sentinel science teams
- Blended products
- Provide useful information for national and
regional management decisions - Applied climatology
- Climate monitoring
- Climate forcings and feedbacks
7Principles of Environmental Data and Information
Stewardship
- Archive and access to fundamental measurements,
products and metadata - CLASS - Data archaeology and improved use CLASS / SDS
- Careful monitoring of observing system
performance for long-term applications - SDS - Generation of authoritative long-term records
through validation of the calibration process,
reprocessing, product generation and the blending
of in situ and satellite measurements - SDS - Provide state of the environment information for
decision makers, and place the current state in
its historical context - SDS
8CLASS Background, Mission, and Overview
9CLASS Background
The CLASS project derives in part from an effort
by NOAA to centralize its numerous systems for
(satellite) data access. The goal of this
effort is to eliminate the various "stove-pipe
systems and produce a unified "enterprise
access system for the NOAA environmental data
holdings.
10CLASS Mission Statement
11CLASS Goals
- Give any potential customer access to
- all NOAA and non-NOAA data
- through a single portal.
- Eliminate the need to keep creating stovepipe
systems for each new type of data, - but, in as much as possible, use already polished
portions/modules - of existing legacy systems.
- Describe a cost-effective architecture that can
primarily handle large-array data sets, - with the capability of handling smaller ones as
well.
12CLASS Overview
- CLASS is a web-based data archive and
distribution system for NOAA/NESDIS environmental
data - Archive ingest, storage, metadata management,
and data quality assurance - Distribution access, visualization, and data
delivery - CLASS is an extension of an 1995 operational
system SAA (Satellite Active Archive) - Transition to the CLASS architecture began in
2001 - CLASS subsumed SAA as the Operational Archive and
Access System for NOAA in 2004 - CLASS currently supports POES, DMSP, and GOES
data sets - CLASS will support additional campaigns, broader
user base, new functionality as it evolves - CLASS concurrently supports ongoing operations
and new requirements implementation
13CLASS Overview (Continued)
- Provide one stop shopping and access capability
for NOAA and NESDIS environmental data and
products - Provide a common look and feel for accessing NOAA
and NESDIS environmental data and products - Provide an efficient architecture for archiving
and distribution of NOAA and NESDIS environmental
data and products - Reduce implementation costs by using
reengineering, and evolutionary effort - Allow NOAA to fulfill its requirements regarding
archive, access and distribution of large array
data sets
14CLASS Overview (Continued)
- Accommodate expanding number of data sources
- MetOp, NPP, NPOESS, EOS, In-situ, NexRAD, GOES-R,
etc. - Data volume is growing exponentially
- Anticipating up to 100 Petabytes by 2015
- User volume is also growing exponentially
15CLASS Synonyms
- Comprehensive Large Array-data Stewardship
System (CLASS) - Archive and Distribution Segment (ADS)
- NPP/NPOESS
- Archive and Access System (AAS)
- METOP/IJPS
- Long-term Archive (LTA)
- Jason
16CLASS as part of the NOAA Observing System
Architecture
CLASS
17CLASS Technical Description Functional Flow
Diagram
Visualize Data
Visualization Data
Data Products and Metadata
Ingest and Store Data
Data Set Inventory
Access Data
Interface with Users
USERS
USERS
USERS
USERS
Data Caches
Data Providers
Data Providers
Data Providers
USERS
Process Orders
Data Providers
Data Providers
Archive
Maintain, Monitor, Control
Orders
CLASS Internet/Intranet
CLASS Operators
18CLASS Technical Description Functional Block
Diagram
19CLASS Project Management
20CLASS Project
- Consortium of NOAA Projects and Budget Lines have
been consolidated under the CLASS Project - CLASS
- Satellite Active Archive (SAA)
- GOES Active Archive (GAA)
- Earth Observing System (EOS)
- Integrated Joint Polar Program (IJPS) / MetOp AAS
- GOES R-series (GOES-R) AAS
- NOAA Virtual Data System (NVDS) and NOAA
E-commerce System (NeS) moved to NCDC
21CLASS Project Plan
- 10 year PAC Plan (Procurement, Acquisition,
Construction) - Road Map for CLASS Program acquisition
- Budgetary Funding Requirements for all CLASS
elements - Life Cycle Planning document
22NOAA DataStewardship Committee (Tom Karl/NCDC)
Archive RequirementsWorking Group (ARWG) (John
Bates/NCDC)
CLASS (Richard G. Reynolds/OSD)
Information Exchange
23CLASS Project Management
Users
NESDIS ITAT
NOAA Data Stewardship Committee
CLASS Project Richard G. Reynolds Charles S.
Bryant
Archive Requirements Working Group (ARWG)
System Engineering Team (SET)
CLASS Project Management Team (CPMT)
System Administration Team (SAT)
CLASS Operations Team (COT)
NGDC Development Teams (Boulder, CO)
OSD/TMC Development Team (Fairmont, WV)
OSD/CSC Development Team (Suitland, MD)
System Integration Test Team (Suitland, MD)
OSDPD-CSC Operations (Suitland, MD)
NCDC-TMC Operations (Asheville, NC)
24CLASS - Process Documentation
CLASS Master Project Management Plan
Procedures
CLASS Software Development Guide
CLASS Configuration Management Plan
CLASS Quality Management Plan
Other Activity Plans
CSC Contract Activity Plan
TMC Contract Activity Plan
25Project Management (Continued)
- CLASS Project Oversight Groups
- Data Stewardship Committee (DSC)
- NESDIS Headquarters / AA and DAA
- NESDIS Information Technology Architecture Team
(ITAT) - - - - - - - - - - -
- CLASS Project Management Team (CPMT)
- Management oversight and coordination
- Risk management
- Project tracking
- Overall decision-making body for CLASS
- System Engineering Team (SET)
- Technical oversight and coordination
- Provides recommendations to CPMT on technical
direction - Provides technical input to CCB
26Project Management (Continued)
- Software Engineering Process Group (SEPG)
- Process definition and improvement
- Provides recommendations to CPMT on process
definition - Configuration Control Board (CCB)
- Change review and control
- Includes CPMT members and SET members
- Systems Administration Team (SAT)
- Develop, Maintain, and Supervise Policies and
Procedures regarding configuration and operation
of CLASS computing resources - CLASS Operations Team (COT)
- Develop and Implement operational policies and
Standard Operating Procedures for the operational
CLASS environments - Includes operations support personnel, system
administrators, database administrators and
archive managers
27CLASSAccomplishments
28CLASS Accomplishments
- Completed overall design of CLASS top-level
architecture - Prepared Key Project and System Documentation
- CLASS Business Case
- System Requirements
- Interface Control Documents (ICDs)
- Concept of Operations (CONOPS)
- Management Plans and Procedures
- Established a project wide risk management
program - CSC Development Team Certified at SEI-CMM Level-3
- Established the CLASS Operations Team (COT)
- Completed Summary 10-year CLASS Project
Budget Requirement - 25M/year
29CLASS Accomplishments (Continued)
- Delivered baseline systems to Suitland and
Asheville - Established Operational Dual-site Configuration
- Established Operational, Integration and Test,
and Development environments in Suitland - Completed migration from SAA to CLASS
- CLASS Operational with POES, DMSP, and GOES data
sets - Completed Preliminary Critical Design Reviews
for the IJPS/Metop Archive and Access Segment - Coordinated with NPP/NPOESS for defining the
IDPS-to-CLASS Interface Control Document - Completed NPP/NPOESS Campaign Implementation
Plan - Worked with NASA personnel to define initial
requirements to archive EOS/MODIS Level-0 data.
30CLASS Accomplishments (Continued)
- Completed Software Release 2.0 / 2.1 / 2.2
- Operational Dual-site configuration -- 02 April
2004 - CLASS Operational with POES, DMSP, and GOES data
sets, plus RadarSat (Synthetic Aperture Radar)
and SeaWiFS (Ocean Color Product) - Completed Software Release 3.0 12 July 2004
- Provides Delivery Manifest and Web Enabled
Subscription Management - Completed Software Release 3.1 October 2005
- Provides Ingest Enhancements to support IJPS
NOAA data - Completed Integration of the Development Teams
- CSC-TMC
- Began SEI/CMMI Certification Process for the
total Development Team
31FY05 CLASS Goals Plans
- Prepare System security Certification
Accreditation (CA) - CLASS currently encompassed by the Satellite
Active Archive CA - Draft stand-alone CLASS CA prepared (March 2005)
- Final to be completed to support relocation to
new NSOF Building in September 2005 - Update the CLASS Long-term Architecture
- Held 2-day architecture session
confirming/upgrading the long-term architectural
plan for CLASS - FY2005 CLASS Long-term Architecture Plan to be
released in September 2005 - Achieve Hardware/Software Commonality among all
Nodes - Project/Development Teams SEI-CMMI Certified
- Initial review completed (January 2005)
- Final assessment to be conducted in May 2005
- Relocate Suitland Node to Boulder (NGDC)
32FY05 CLASS Goals Plans (Continued)
- METOP-1 Pre-Launch Testing and Operational
Readiness (Completed March 2005) - Capability contained in Release-3.2
- Awaiting data flows from the Spacecraft for final
functional verification - NPP Campaign Development and Testing
- EOS-MODIS Campaign Development and Testing
- Establish an interface with NeS
- Establish an interface with NMMR
- Metadata Campaign development continues
- Geospatial Capability development begins
- Jason/OSTM Campaign development begins
- Establish a Development Environment at
TMC/Fairmont - ----------------
- Operations Continue
33FY05 Hardware/Software Plans
- System SAN Capacity Upgrade (Completed January
2005) - Additional disk space at both CLASS operational
sites - Data Direct Networks 56 Tbytes (expandable to
302 Tbytes) - CLASS Release 3.2 (Completed April 2005)
- Support for Metop-1 data / readiness for IJPS
End-to-End test - Subscription for GOES data w/ separate GVAR data
families GOES-N - Upgrade all systems to AIX 5.1/5.2 (64-bit
structures) - CLASS Release 3.3 (Scheduled for July 2005)
- McIdas-less ingest
- Upgrades to the Help Pages/Static Pages
- Map server upgrades Point Searches
- CLASS-NMMR Interface
- Security enhancements, including capability to
deliver data encrypted - UTC Time utilization
34FY06 CLASS Goals Plans
- METOP-1 Operational Activation
- NPP Campaign End-to-End Testing, Compatibility
Testing, and Operational Readiness - Metadata Campaign development continues
- Geospatial Capability development continues
- Jason/OSTM Campaign Testing
- Data QA/QC Campaign begins
- Reprocessing Campaign begins
35FY06 Hardware/Software Plans
- System Storage Capacity Upgrade
- Scheduled for September 2006
- LTO-2 to LTO-3 Migration
- CLASS Release 4.0 (Scheduled for October 2005)
- Basic NPP Support
- Final IJPS/Metop Pre-launch Release
- CLASS NeS Interface
- CLASS NMMR Interface
- CLASS Release 4.1 (Scheduled for February 2006)
- NPP Readiness for NCT-3
- Initial Data Delivery Upgrades
- CLASS Release 4.2 (Scheduled for August 2006)
- NPP Readiness for NCT-4
- CLASS Release 4.3 (Scheduled for October 2006)
- NPP Pre-Launch Release
- Complete Data Delivery Upgrades
36CLASS Statistics(Average Last 12 Months)
- Ingest 71 GB/Day 26 TB/Year
- Ingest 860,000 Data Sets/Year
- Distribution (On-line Subscriptions) - 44
TB/Year . 3.63 TB/Month - Distribution (On-line Subscriptions) -
3,170,000 Data Sets/Year 263,888 Data
Sets/Month
37CLASS BUDGETSand Campaigns
Budget numbers are shown for the purpose of
establishing a reference for relative complexity
of a requirement and level of completeness, and
do not represent NOAA, Department of Commence, or
The Presidents position regarding specific
Congressional Budget Requests.
38Major Core-CLASS ProjectFunctional Campaigns
- Core CLASS Baseline System Development,
Expansion, Evolution - FY04-FY16 94M
- Metadata Campaign
- FY04-FY14 12M
- Reprocessing Campaign
- FY06-FY16 35M
- QA/QC
- FY06 . 2M/year
- --------------------
- System OM
- FY04 (2M)-FY14(10M) 11M/year thereafter
39General Core-CLASS Activities
- Support for CLASS Development Activities
- Architecture Design
- System Design
- Software Development and Integration
- Quality Assurance
- System Engineering Team Activities
- Support for CLASS Operations and Maintenance
- Contractor and Government Project Support
- Licenses
- Hardware Maintenance
- Hardware Refresh
40General Core-CLASS Activities (Continued)
- Metadata
- Assure FGDC Compliance
- Establish Rich Metadata Standards
- Develop Integrated data and metadata access
- Geospatial
- Capability development and implementation
- Integration of all CLASS data
- Web
- Integrate SABR into CLASS
- Design and implement web mapping system
- Enhancements to support Data Mining capability
- Data Mining
- Architecture design and implementation
- Application design and implementation
- Reprocessing
- Architecture design and implementation
- Establish operational processes and standards
41Major CLASS ProjectData Campaigns
- Metop-1
- FY01-FY07 6.5M
- NPP
- FY04-FY11 15.8M
- EOS-MODIS
- FY04-FY10 16.7M
- NDE
- FY06-FY12 1.5M
- NEXRAD
- FY06-FY09 8.1M
- Insitu
- FY06-FY14 8.0M
- EOS Retrospective
- FY07-FY13 8.2M
- GOES-R
- FY07-FY14 41M
- Metop-2
- FY08-FY14 5.0M
- NPOESS-C1
- FY08-FY12 11.1M
42General Data Campaign Requirements
- Requirements Definition and ICDs
- Data and Products Ingest
- Storage, Processing, and Communications Upgrades
- Metadata extensions
- Catalogue extensions
- Visualization extensions
- Reprocessing extensions
43CLASS Budgets
- FY01 1.995M
- FY02 3.599M
- FY03 2.881M
----------- - FY04 10.5M
- FY05 14.6M
- FY06 11.9M
- FY07 18.0M
- FY08 20.8M
- FY09 25.5M
- FY10 29.3M/yr
- FY07 PAE Recommendations
-
- FY05 Funding Sources
- CLASS 6.6M
- GAA 2.5M
- EOS 3.0M
- SAA 1.5M
- OSD 1.0M
POES Ground Systems PAC METOP/IJPS Archive
Access 200K CLASS Hardware 800K
44FY06-07 CLASS Budgets
- FY06 11.9M
- CLASS 6.6M
- SAA 1.5M
- EOS 3.0M
- OSD 0.8M
- FY07 20.0M
- CLASS 15.1M
- SAA 1.5M
- EOS 1.0M
- GOES-R 2.0M
- OSD 0.4M
45CLASSFunctionality and User Services
46Core Functionality
- Data and metadata ingest, archive, and
replication (NARA compliant) - Delivery Manifests
- Data Quality
- Companion Files
- Metadata Management
- User Data Discovery
- Access, Search, and Visualization
47Core Functionality (Continued)
- User Access
- Web
- Single Sign-on/On-line
- Subscription
- Machine-to-Machine/Bulk Order
- User Search
- Inventory
- Temporal /Spatial
- Data Type
- Metadata Characteristics
- User Visualization
- Imagery Chips
- Geospatial Navigation
- Overlays
48Core Functionality (Continued)
- Data Delivery System
- Improved use of LTO tape robotic system
capabilities - Consolidate multiple order delivery systems
(Online, Subscription, Bulk Order) - Improved cache management
- Delivery Options
- Physical media creation and shipment
- Electronic delivery
- E-Commerce (NESDIS E-commerce System NeS)
- Pricing, Payment receipt, Payment processing
- User Profiles
- Data Denial
49Core Functionality (Continued)
- Data Delivery to the User
- Format options
- Push
- Pull
- Physical Media
- Delivery Notifications
- User Helps
- New Requirements via the ARWG!
50THANK YOU!