CSR Charts

1
GLAST Large Area Telescope ISOC Peer Review
Section 4.3 Pipeline, Data Storage and
Networking Issues for the ISOC Richard
Dubois SAS System Manager
2
Outline

• SAS Summary Requirements
• Pipeline
• Requirements
• Processing database
• Prototype status
• Networking
• Proposed Network Topology
• Network Monitoring
• File exchange
• Security
• Data Storage and Archive

3
Level III Requirements Summary
Ref LAT-SS-00020

• Basic requirements are to routinely handle 10
  GB/day in multiple passes coming from the MOC
• _at_ 150 Mb/s 2 GB should take lt 2 mins
• Outgoing volume to SSC is ltlt 1 GB/day
• NASA 2810 Security Regs normal security levels
  for IOCs as practiced by computing centers
  already

4
Pipeline Spec

• Function
• The Pipeline facility has five major functions
• automatically process Level 0 data through
  reconstruction (Level 1) 
• provide near real-time feedback to IOC 
• facilitate the verification and generation of new
  calibration constants 
• produce bulk Monte Carlo simulations 
• backup all data that passes through
• Must be able to perform these functions in
  parallel
• Fully configurable, parallel task chains allow
  great flexibility for use online as well as
  offline
• Will test the online capabilities during Flight
  Integration
• The pipeline database and server, and diagnostics
  database have been specified (will need revision
  after prototype experience!)
• database LAT-TD-00553
• server LAT-TD-00773
• diagnostics LAT-TD-00876

5
Expected Capacity

• We routinely made use of 100-300 processors on
  the SLAC farm for repeated Monte Carlo
  simulations, lasting weeks
• Expanding farm net to France and Italy
• Unknown yet what our MC needs will be
• We are very small compared to our SLAC neighbour
  BABAR computing center sized for them
• 2000-3000 CPUS 300 TB of disk 6 robotic silos
  holding 30000 200 GB tapes total
• SLAC computing center has guaranteed our needs
  for CPU and disk, including maintenance for the
  life of the mission.
• Data rate small compared to already demonstrated
  MC capability
• 75 of todays CPUs to handle 5 hrs of data in 1
  hour _at_ 0.15 sec/event
• Onboard compression may make it 75 of tomorrows
  CPUs too

6
Pipeline in Pictures
State machine complete processing record
Expandable and configurable set of processing
nodes
Configurable linked list of applications to run
7
Processing Dataset Catalogue
Datasets grouped by task
Processing records
Datasets info is here
8
First Prototype - OPUS
Open source project from STScI In use by several
missions Now outfitted to run DC1 dataset
OPUS Java mangers for pipelines
9
Disk and Archives

• We expect 10 GB raw data per day and assume
  comparable volume of events for MC
• Leads to 50 TB over 5 years for all data types
• No longer frightening keep it all on disk
• Use SLACs mstore archiving system to keep a copy
  in the silo
• Already practicing with it and will hook it up to
  OPUS
• Archive all data we touch track in dataset
  catalogue
• Not an issue

10
Network Path SLAC-Goddard
?
?
?
?
?
SLAC Stanford Oakland (CENIC)
LA UC-AID (Abilene) Houston
Atlanta Washington GSFC
(77 ms ping)
11
ISOC Stanford/SLAC Network

• SLAC Computing Center
• OC48 connection to outside world
• provides data connections to MOC and SSC
• hosts the data and processing pipeline
• Transfers MUCH larger datasets around the world
  for BABAR
• World renowned for network monitoring expertise
• Will leverage this to understand our open
  internet model
• Sadly, a great deal of expertise with enterprise
  security as well
• Part of ISOC expected to be in new Kavli
  Institute building on campus
• Connected by fiber (2 ms ping)
• Mostly monitoring and communicating with
  processes/data at SLAC

12
Network Monitoring
Need to understand failover reliability, capacity
and latency
13
LAT Monitoring
LAT Monitoring Keep track of connections to
collaboration sites Alerts if they go
down Fodder for complaints if poor connectivity
Monitoring nodes at most LAT collaborating
institutions
14
File Exchange

• Secure
• No passwords in plain text etc
• Reliable
• Has to work gt 99 of the time (say)
• handle the (small) data volume
• order 10 GB/day from Goddard (MOC) 0.3 GB/day
  back to Goddard (SSC)
• keep records of transfers
• database records of files sent and received
• handshakes
• both ends agree on what happened
• some kind of clean error recovery
• Notification sent out on failures
• Web interface to track performance
• Are investigating DTS now
• Not super happy with how its built as far as
  reliability and security go
• Installing it now
• Starting to work with author on issues

15
Security

• Network security application vs network
• ssh/vpn among all sites MOC, SSC and internal
  ISOC
• A possible avenue is to make all applications
  secure (ie encrypted), using SSL.
• File and Database security
• Controlled membership in disk ACLs
• Controlled access to databases
• Depend on SLAC security otherwise

16
Summary

• We are testing out the OPUS pipeline as our first
  prototype
• Already outfitted to run DC1 dataset
• Interfaces to processing database and SLAC batch
  done
• Practice with DCs and Flight Integration
• We expect to need O(50 TB) of disk and 2-3x that
  in tape archive
• Not an issue
• We expect to use Internet2 connectivity for
  reliable and fast transfer of data between SLAC
  and Goddard
• Transfer rates of gt 150 Mb/s already demonstrated
• lt 2 min transfer for standard downlink. More than
  adequate.
• Starting a program of routine network monitoring
  to practice
• Network security is an ongoing, but largely
  solved, problem
• There are well-known mechanisms to protect sites
• We will leverage considerable expertise from the
  SLAC and Stanford networking/security folks