Comprehensive Large Arraydata Stewardship System CLASS http:www.class.noaa.gov

1
Comprehensive Large Array-data Stewardship System
(CLASS) http//www.class.noaa.gov

• What is CLASS and How Will It Serve NOAA
  Clientele?
• 12 May 2005
• Richard G. Reynolds / CLASS Project Manager

2
Agenda

• Data Centers and Data Stewardship
• CLASS System Introduction
• Project Management
• Accomplishments and Near-term Goals
• Budgets and Campaigns
• Functionality User Services

3
NOAAs National Data Centers

• NOAAs National Data Centers are major archive,
  access, and assessment sites maintaining,
  processing, and distributing environmental and
  geospatial data.
• National Climatic Data Center
  WWW.NCDC.NOAA.GOV
• Asheville, NC
• National Coastal Data Development Center
  WWW.NCDDC.NOAA.GOV
• Stennis, MS
• National Geophysical Data Center
  WWW.NGDC.NOAA.GOV
• Boulder, CO
• National Oceanographic Data Center
  WWW.NODC.NOAA.GOV
• Silver Spring, MD

4
NOAAs National Data Centers(Continued)

• These Centers provide long-term stewardship for
  most of NOAAs environmental and geospatial data,
  and a broad range of user services.
• Centers of data are facilities where extensive
  collections of given environmental parameter(s)
  are maintained because of individual or
  institutional research or operational
  requirements
• They also serve as Agency Record Centers

5
NOAAs National Data Centersare Environmental
Data Stewards
Scientific Data Stewardship is ownership,
knowledge, utilization, and application of the
data CLASS is the Information Technology
infrastructure (hardware and software
environment, and tools) underpinning SDS Data
Rescue preserves and makes available historical
data sets from obsolete media
6
Scientific Data Stewardship (SDS)

• Observing system performance monitoring
• Bias monitoring
• Data character and metadata
• Climate Data Records (CDRs) transition from
  research to products and services
• Sentinel science teams
• Blended products
• Provide useful information for national and
  regional management decisions
• Applied climatology
• Climate monitoring
• Climate forcings and feedbacks

7
Principles of Environmental Data and Information
Stewardship

• Archive and access to fundamental measurements,
  products and metadata - CLASS
• Data archaeology and improved use CLASS / SDS
• Careful monitoring of observing system
  performance for long-term applications - SDS
• Generation of authoritative long-term records
  through validation of the calibration process,
  reprocessing, product generation and the blending
  of in situ and satellite measurements - SDS
• Provide state of the environment information for
  decision makers, and place the current state in
  its historical context - SDS

8
CLASS Background, Mission, and Overview
9
CLASS Background
The CLASS project derives in part from an effort
by NOAA to centralize its numerous systems for
(satellite) data access. The goal of this
effort is to eliminate the various "stove-pipe
systems and produce a unified "enterprise
access system for the NOAA environmental data
holdings.
10
CLASS Mission Statement
11
CLASS Goals

• Give any potential customer access to
• all NOAA and non-NOAA data
• through a single portal.
• Eliminate the need to keep creating stovepipe
  systems for each new type of data,
• but, in as much as possible, use already polished
  portions/modules
• of existing legacy systems.
• Describe a cost-effective architecture that can
  primarily handle large-array data sets,
• with the capability of handling smaller ones as
  well.

12
CLASS Overview

• CLASS is a web-based data archive and
  distribution system for NOAA/NESDIS environmental
  data
• Archive ingest, storage, metadata management,
  and data quality assurance
• Distribution access, visualization, and data
  delivery
• CLASS is an extension of an 1995 operational
  system SAA (Satellite Active Archive)
• Transition to the CLASS architecture began in
  2001
• CLASS subsumed SAA as the Operational Archive and
  Access System for NOAA in 2004
• CLASS currently supports POES, DMSP, and GOES
  data sets
• CLASS will support additional campaigns, broader
  user base, new functionality as it evolves
• CLASS concurrently supports ongoing operations
  and new requirements implementation

13
CLASS Overview (Continued)

• Provide one stop shopping and access capability
  for NOAA and NESDIS environmental data and
  products
• Provide a common look and feel for accessing NOAA
  and NESDIS environmental data and products
• Provide an efficient architecture for archiving
  and distribution of NOAA and NESDIS environmental
  data and products
• Reduce implementation costs by using
  reengineering, and evolutionary effort
• Allow NOAA to fulfill its requirements regarding
  archive, access and distribution of large array
  data sets

14
CLASS Overview (Continued)

• Accommodate expanding number of data sources
• MetOp, NPP, NPOESS, EOS, In-situ, NexRAD, GOES-R,
  etc.
• Data volume is growing exponentially
• Anticipating up to 100 Petabytes by 2015
• User volume is also growing exponentially

15
CLASS Synonyms

• Comprehensive Large Array-data Stewardship
  System (CLASS)
• Archive and Distribution Segment (ADS)
• NPP/NPOESS
• Archive and Access System (AAS)
• METOP/IJPS
• Long-term Archive (LTA)
• Jason

16
CLASS as part of the NOAA Observing System
Architecture
CLASS
17
CLASS Technical Description Functional Flow
Diagram
Visualize Data
Visualization Data
Data Products and Metadata
Ingest and Store Data
Data Set Inventory
Access Data
Interface with Users
USERS
USERS
USERS
USERS
Data Caches
Data Providers
Data Providers
Data Providers
USERS
Process Orders
Data Providers
Data Providers
Archive
Maintain, Monitor, Control
Orders
CLASS Internet/Intranet
CLASS Operators
18
CLASS Technical Description Functional Block
Diagram
19
CLASS Project Management
20
CLASS Project

• Consortium of NOAA Projects and Budget Lines have
  been consolidated under the CLASS Project
• CLASS
• Satellite Active Archive (SAA)
• GOES Active Archive (GAA)
• Earth Observing System (EOS)
• Integrated Joint Polar Program (IJPS) / MetOp AAS
• GOES R-series (GOES-R) AAS
• NOAA Virtual Data System (NVDS) and NOAA
  E-commerce System (NeS) moved to NCDC

21
CLASS Project Plan

• 10 year PAC Plan (Procurement, Acquisition,
  Construction)
• Road Map for CLASS Program acquisition
• Budgetary Funding Requirements for all CLASS
  elements
• Life Cycle Planning document

22
NOAA DataStewardship Committee (Tom Karl/NCDC)
Archive RequirementsWorking Group (ARWG) (John
Bates/NCDC)
CLASS (Richard G. Reynolds/OSD)
Information Exchange
23
CLASS Project Management
Users
NESDIS ITAT
NOAA Data Stewardship Committee
CLASS Project Richard G. Reynolds Charles S.
Bryant
Archive Requirements Working Group (ARWG)
System Engineering Team (SET)
CLASS Project Management Team (CPMT)
System Administration Team (SAT)
CLASS Operations Team (COT)
NGDC Development Teams (Boulder, CO)
OSD/TMC Development Team (Fairmont, WV)
OSD/CSC Development Team (Suitland, MD)
System Integration Test Team (Suitland, MD)
OSDPD-CSC Operations (Suitland, MD)
NCDC-TMC Operations (Asheville, NC)
24
CLASS - Process Documentation
CLASS Master Project Management Plan
Procedures
CLASS Software Development Guide
CLASS Configuration Management Plan
CLASS Quality Management Plan
Other Activity Plans
CSC Contract Activity Plan
TMC Contract Activity Plan
25
Project Management (Continued)

• CLASS Project Oversight Groups
• Data Stewardship Committee (DSC)
• NESDIS Headquarters / AA and DAA
• NESDIS Information Technology Architecture Team
  (ITAT)
• - - - - - - - - - -
• CLASS Project Management Team (CPMT)
• Management oversight and coordination
• Risk management
• Project tracking
• Overall decision-making body for CLASS
• System Engineering Team (SET)
• Technical oversight and coordination
• Provides recommendations to CPMT on technical
  direction
• Provides technical input to CCB

26
Project Management (Continued)

• Software Engineering Process Group (SEPG)
• Process definition and improvement
• Provides recommendations to CPMT on process
  definition
• Configuration Control Board (CCB)
• Change review and control
• Includes CPMT members and SET members
• Systems Administration Team (SAT)
• Develop, Maintain, and Supervise Policies and
  Procedures regarding configuration and operation
  of CLASS computing resources
• CLASS Operations Team (COT)
• Develop and Implement operational policies and
  Standard Operating Procedures for the operational
  CLASS environments
• Includes operations support personnel, system
  administrators, database administrators and
  archive managers

27
CLASSAccomplishments
28
CLASS Accomplishments

• Completed overall design of CLASS top-level
  architecture
• Prepared Key Project and System Documentation
• CLASS Business Case
• System Requirements
• Interface Control Documents (ICDs)
• Concept of Operations (CONOPS)
• Management Plans and Procedures
• Established a project wide risk management
  program
• CSC Development Team Certified at SEI-CMM Level-3
• Established the CLASS Operations Team (COT)
• Completed Summary 10-year CLASS Project
  Budget Requirement - 25M/year

29
CLASS Accomplishments (Continued)

• Delivered baseline systems to Suitland and
  Asheville
• Established Operational Dual-site Configuration
• Established Operational, Integration and Test,
  and Development environments in Suitland
• Completed migration from SAA to CLASS
• CLASS Operational with POES, DMSP, and GOES data
  sets
• Completed Preliminary Critical Design Reviews
  for the IJPS/Metop Archive and Access Segment
• Coordinated with NPP/NPOESS for defining the
  IDPS-to-CLASS Interface Control Document
• Completed NPP/NPOESS Campaign Implementation
  Plan
• Worked with NASA personnel to define initial
  requirements to archive EOS/MODIS Level-0 data.

30
CLASS Accomplishments (Continued)

• Completed Software Release 2.0 / 2.1 / 2.2
• Operational Dual-site configuration -- 02 April
  2004
• CLASS Operational with POES, DMSP, and GOES data
  sets, plus RadarSat (Synthetic Aperture Radar)
  and SeaWiFS (Ocean Color Product)
• Completed Software Release 3.0 12 July 2004
• Provides Delivery Manifest and Web Enabled
  Subscription Management
• Completed Software Release 3.1 October 2005
• Provides Ingest Enhancements to support IJPS
  NOAA data
• Completed Integration of the Development Teams
• CSC-TMC
• Began SEI/CMMI Certification Process for the
  total Development Team

31
FY05 CLASS Goals Plans

• Prepare System security Certification
  Accreditation (CA)
• CLASS currently encompassed by the Satellite
  Active Archive CA
• Draft stand-alone CLASS CA prepared (March 2005)
• Final to be completed to support relocation to
  new NSOF Building in September 2005
• Update the CLASS Long-term Architecture
• Held 2-day architecture session
  confirming/upgrading the long-term architectural
  plan for CLASS
• FY2005 CLASS Long-term Architecture Plan to be
  released in September 2005
• Achieve Hardware/Software Commonality among all
  Nodes
• Project/Development Teams SEI-CMMI Certified
• Initial review completed (January 2005)
• Final assessment to be conducted in May 2005
• Relocate Suitland Node to Boulder (NGDC)

32
FY05 CLASS Goals Plans (Continued)

• METOP-1 Pre-Launch Testing and Operational
  Readiness (Completed March 2005)
• Capability contained in Release-3.2
• Awaiting data flows from the Spacecraft for final
  functional verification
• NPP Campaign Development and Testing
• EOS-MODIS Campaign Development and Testing
• Establish an interface with NeS
• Establish an interface with NMMR
• Metadata Campaign development continues
• Geospatial Capability development begins
• Jason/OSTM Campaign development begins
• Establish a Development Environment at
  TMC/Fairmont
• ----------------
• Operations Continue

33
FY05 Hardware/Software Plans

• System SAN Capacity Upgrade (Completed January
  2005)
• Additional disk space at both CLASS operational
  sites
• Data Direct Networks 56 Tbytes (expandable to
  302 Tbytes)
• CLASS Release 3.2 (Completed April 2005)
• Support for Metop-1 data / readiness for IJPS
  End-to-End test
• Subscription for GOES data w/ separate GVAR data
  families GOES-N
• Upgrade all systems to AIX 5.1/5.2 (64-bit
  structures)
• CLASS Release 3.3 (Scheduled for July 2005)
• McIdas-less ingest
• Upgrades to the Help Pages/Static Pages
• Map server upgrades Point Searches
• CLASS-NMMR Interface
• Security enhancements, including capability to
  deliver data encrypted
• UTC Time utilization

34
FY06 CLASS Goals Plans

• METOP-1 Operational Activation
• NPP Campaign End-to-End Testing, Compatibility
  Testing, and Operational Readiness
• Metadata Campaign development continues
• Geospatial Capability development continues
• Jason/OSTM Campaign Testing
• Data QA/QC Campaign begins
• Reprocessing Campaign begins

35
FY06 Hardware/Software Plans

• System Storage Capacity Upgrade
• Scheduled for September 2006
• LTO-2 to LTO-3 Migration
• CLASS Release 4.0 (Scheduled for October 2005)
• Basic NPP Support
• Final IJPS/Metop Pre-launch Release
• CLASS NeS Interface
• CLASS NMMR Interface
• CLASS Release 4.1 (Scheduled for February 2006)
• NPP Readiness for NCT-3
• Initial Data Delivery Upgrades
• CLASS Release 4.2 (Scheduled for August 2006)
• NPP Readiness for NCT-4
• CLASS Release 4.3 (Scheduled for October 2006)
• NPP Pre-Launch Release
• Complete Data Delivery Upgrades

36
CLASS Statistics(Average Last 12 Months)

• Ingest 71 GB/Day 26 TB/Year
• Ingest 860,000 Data Sets/Year
• Distribution (On-line Subscriptions) - 44
  TB/Year . 3.63 TB/Month
• Distribution (On-line Subscriptions) -
  3,170,000 Data Sets/Year 263,888 Data
  Sets/Month

37
CLASS BUDGETSand Campaigns
Budget numbers are shown for the purpose of
establishing a reference for relative complexity
of a requirement and level of completeness, and
do not represent NOAA, Department of Commence, or
The Presidents position regarding specific
Congressional Budget Requests.
38
Major Core-CLASS ProjectFunctional Campaigns

• Core CLASS Baseline System Development,
  Expansion, Evolution
• FY04-FY16 94M
• Metadata Campaign
• FY04-FY14 12M
• Reprocessing Campaign
• FY06-FY16 35M
• QA/QC
• FY06 . 2M/year
• --------------------
• System OM
• FY04 (2M)-FY14(10M) 11M/year thereafter

39
General Core-CLASS Activities

• Support for CLASS Development Activities
• Architecture Design
• System Design
• Software Development and Integration
• Quality Assurance
• System Engineering Team Activities
• Support for CLASS Operations and Maintenance
• Contractor and Government Project Support
• Licenses
• Hardware Maintenance
• Hardware Refresh

40
General Core-CLASS Activities (Continued)

• Metadata
• Assure FGDC Compliance
• Establish Rich Metadata Standards
• Develop Integrated data and metadata access
• Geospatial
• Capability development and implementation
• Integration of all CLASS data
• Web
• Integrate SABR into CLASS
• Design and implement web mapping system
• Enhancements to support Data Mining capability
• Data Mining
• Architecture design and implementation
• Application design and implementation
• Reprocessing
• Architecture design and implementation
• Establish operational processes and standards

41
Major CLASS ProjectData Campaigns

• Metop-1
• FY01-FY07 6.5M
• NPP
• FY04-FY11 15.8M
• EOS-MODIS
• FY04-FY10 16.7M
• NDE
• FY06-FY12 1.5M
• NEXRAD
• FY06-FY09 8.1M

• Insitu
• FY06-FY14 8.0M
• EOS Retrospective
• FY07-FY13 8.2M
• GOES-R
• FY07-FY14 41M
• Metop-2
• FY08-FY14 5.0M
• NPOESS-C1
• FY08-FY12 11.1M

42
General Data Campaign Requirements

• Requirements Definition and ICDs
• Data and Products Ingest
• Storage, Processing, and Communications Upgrades
• Metadata extensions
• Catalogue extensions
• Visualization extensions
• Reprocessing extensions

43
CLASS Budgets

• FY01 1.995M
• FY02 3.599M
• FY03 2.881M
  -----------
• FY04 10.5M
• FY05 14.6M
• FY06 11.9M
• FY07 18.0M
• FY08 20.8M
• FY09 25.5M
• FY10 29.3M/yr
• FY07 PAE Recommendations

• FY05 Funding Sources
• CLASS 6.6M
• GAA 2.5M
• EOS 3.0M
• SAA 1.5M
• OSD 1.0M

POES Ground Systems PAC METOP/IJPS Archive
Access 200K CLASS Hardware 800K
44
FY06-07 CLASS Budgets

• FY06 11.9M
• CLASS 6.6M
• SAA 1.5M
• EOS 3.0M
• OSD 0.8M

• FY07 20.0M
• CLASS 15.1M
• SAA 1.5M
• EOS 1.0M
• GOES-R 2.0M
• OSD 0.4M

45
CLASSFunctionality and User Services
46
Core Functionality

• Data and metadata ingest, archive, and
  replication (NARA compliant)
• Delivery Manifests
• Data Quality
• Companion Files
• Metadata Management
• User Data Discovery
• Access, Search, and Visualization

47
Core Functionality (Continued)

• User Access
• Web
• Single Sign-on/On-line
• Subscription
• Machine-to-Machine/Bulk Order
• User Search
• Inventory
• Temporal /Spatial
• Data Type
• Metadata Characteristics
• User Visualization
• Imagery Chips
• Geospatial Navigation
• Overlays

48
Core Functionality (Continued)

• Data Delivery System
• Improved use of LTO tape robotic system
  capabilities
• Consolidate multiple order delivery systems
  (Online, Subscription, Bulk Order)
• Improved cache management
• Delivery Options
• Physical media creation and shipment
• Electronic delivery
• E-Commerce (NESDIS E-commerce System NeS)
• Pricing, Payment receipt, Payment processing
• User Profiles
• Data Denial

49
Core Functionality (Continued)

• Data Delivery to the User
• Format options
• Push
• Pull
• Physical Media
• Delivery Notifications
• User Helps
• New Requirements via the ARWG!

50
THANK YOU!