ISOC Status Review

1
ISOC Status Review

• June 3, 2004

2
Overview

• We have developed an organization and staffing
  plan in concert with the SLAC management.
• ISOC buildup started, rapid ramp up over next
  year
• We have completed initial work on an operations
  architecture.
• We have made good progress in addressing peer
  review RFAs
• Substantial work remains before CDR but we
  believe we now understand the scope and will be
  ready by 7/15.

3
LAT/ISOC Organization Post Launch
SSAC
NASA GLAST Project
Science Analysis Coordination Committee SAC
head, Analysis leads, ISOC rep, SAS head
4.1
PI P. Michelson Inst Sci S.Ritz
Instrument Ops Advisory Board H/W subsystem
leads, key Technical Advisors from throughout
collaboration
4.1.B
4.1.B.4
ISOC Manager W. Craig (Acting)
Science Analysis Center
Resources Only
4.1.B.1
4.1.B.2
4.1.B.3
LAT Ops Facility (LOF)
Sci. Ops Group (SOG)
Sci. Analysis SW (SAS) R. Dubois
Software
Calib
Flt S/w Testbed
Pipeline
Collab
Operations
Optimization
Pipeline Config.
Anayl Tools
Computing
4
Staffing

• Rob Cameron has accepted the ISOC manager
  position, so there will (finally) be a permanent
  ISOC manager in place in August.
• Craig will be responsible for a successful CDR
  and will keep Cameron updated throughout.
• Several month transition period planned
• Steve Culp has accepted S/W developer position
  and will start within a week. He will be
  responsible for fleshing out the architecture and
  first database implementations.

5
Staffing Profiles
Excludes SAS and SAC.
6
Staffing Profiles (with SAS/SAC)
Does not include Stanford, UCSC, NRL, GSFC or
collaboration members.
7
Architecture

• Drivers
• Minimize VV burden and total cost
• Maintain all science capabilities
• Simplify interfaces and allow early testing
• Recognized that neither of the previously
  considered options were particularly attractive
• ITOS/Commercial packages dont accommodate
  complexities of science data
• Homegrown system doesnt have heritage, not ready
  in time to make project timelines.
• Most of additional code needed duplicates that in
  existing packages
• ? Studied hybrid solutions

8
ITOS/Astro RT Trade

• In favor of either
• Both AstroRT and ITOS would provide basic
  instrument health and safety functions
• Telemetry display
• EU conversion
• Limit checking and monitoring
• Trending
• Command and telemetry database access
• Both products have learnable interfaces and
  scripting
• AstroRT uses LabView for display and Perl scripts
  for automation
• ITOS displays are reportedly easy to create, uses
  STOL for input

9
ITOS/Astro RT Trade

• Against either
• Requires use of ITOS or Astro-RT specific
  interfaces and scripting
• Both have ITAR issues
• Limitations are not fully understood
• Believe limitations will not affect monitoring
  and trending of housekeeping data only science
  and instrument diagnostics

10
ITOS/Astro RT Trade

• In favor of AstroRT
• LAT is using AstroRT for LAT flight software
  testing
• Against AstroRT
• Does not handle character strings not sure if
  thats an issue for us (it is with GBM)
• Commercial product costing upfront and for
  support throughout program life
• Probably unable to alter AstroRT code
• In favor of ITOS
• MOC and GBM will be using ITOS
• May be able to alter ITOS code or have changes
  made
• Against ITOS
• None that dont also exist for AstroRT

11
Proposed ISOC S/W Architecture
No reqts on MOC that require LATOPS layer
All State of Health requirements satisfied within
ITOS
MOC/GSSC
ITOS
Cmd DB
SOH trending and display
Data
Cmd
LATTE Ops ? LATOPS
Science data/performance trending Register load
generation
Relational database interaction Pipeline/SAS
interactions
12
RFA responses
RFA Summary Requestor Actionee Comment
1 a. Need ISOC Management Plan Approachb. ISOC Documentation Set R. Schweiss W. Craig Plan draft and list of ISOC documents on http//www-glast.slac.stanford.edu/ioc/
2 Need overall functional block diagram illustrating the functional capabilities and data flow during various phases R. Schweiss L. Bator Draft response slides attached
3 Risk Analysis R. Schweiss W. Craig Draft response slides attached
4 Reschedule ISOC CDR M. RackleyC. Young D. Lung Done. CDR scheduled for 8/4/04
5 Incomplete Level III requirements for LOF and SOG M. Rackley L. Bator Drafts on http//www-glast.slac.stanford.edu/ioc/
6 Staffing plan and profile M. RackleyC. Young W. CraigD. Lung Staffing plan and profile presented, RFA response pending
7 Define the ISOC reports for internal use and external use M. Rackley L. Bator Response complete slides attached
8 The ISOC does not yet know what system it is using to process Observatory HSK data or perform the commanding M. Rackley L. Bator Architecture presented, RFA response pending
9 Describe lesson learned approach M. Rackley W. Craig Response complete slides attached
13
RFA responses, contd
RFA Summary Requestor Actionee Comment
10 ISOC verification does not involve early opportunities to validate/test using LAT instrument M. Rackley N. Johnson L. Bator See RFA 2, also pending architecture approval
11 Verify LAT modes M. Davis L. Bator Draft response slides attached
12 Understand the number of writes to EEPROM C. Young L. Bator Response submitted
13 ISOC detailed development schedule K. Lehtonen D. Lung Pending architecture approval
14 Enter a more formal agreement with SLAC management on required data storage and processing requirements N. Johnson W. Craig Response completed slides attached
15 ISOC organization structure communications N. Johnson W. CraigD. Lung Organization presented, RFA response pending
16 Define mechanism for ISOC requirements being placed on IT and SAS N. Johnson W. Craig Pending architecture approval
17 Define LOF/SOG tools R. Corbet L. BatorJ. Panetta Draft response slides attached
18 Specify plans and requirements for automation of Ops software R. CorbetM. Rackley L. BatorJ. Panetta Draft response for 1st part, awaiting S. Culp for 2nd
19 Specify plans and requirements for Ops SW to be of sufficient robustness R. Corbet L. BatorJ. Panetta ECD 6/15/04 S. Culp
20 Specify what other ground system elements will be involved in LAT operations R. Corbet L. BatorD. Lung ECD 7/5/04 - working group on contingency plans
14
RFA 2 ISOC Functional Block Diagram

• RFA 2 Specific Request
• Need an overall functional block diagram
  illustrating the functional capabilities and a
  data flow diagram showing the various data flows,
  with the differences among the IT (pre-launch
  w/GSE) phase, LEO phase, and nominal on-orbit
  phase configurations specified
• Diagrams for each phase might be needed

15
ISOC Dataflow During IT Single Tower Testing

• Obtain data during IT EM2 testing
• Goal is to read houskeeping data off flat file
  produced by Online
• Database development and maintenance is shared
  between IT and ISOC

16
ISOC Dataflow During IT Multi-Tower Testing

• Obtain data during IT testing
• Increase in ISOC functionality

17
ISOC Dataflow with TestBed - Direct to SIU

• Direct interface with SIU for CCSDS command and
  telemetry packets
• Obtain testbed simulated data via SIU
• Demonstration of ISOC capability increases as
  functionality is developed

18
ISOC Dataflow with TestBed - With SIIS

• Interface with SIIS/AstroRT for telemetry packets
  and commanding
• Obtain testbed simulated data via SIU and SIIS
• Demonstration of ISOC capability increases as
  functionality is developed

19
ISOC Dataflow During GRTs, LEO and On-orbit

• Shows full ISOC capability for LEO and On-orbit
• GRTs will test capabilities as they are available

20
RFA 3 - ISOC Risk Analysis

• Process
• Discussion with IT personnel on risks
• Internal discussion performed in concert with
  RFAs from peer review
• Review and approval by ISOC stakeholders
• Follow-up
• Entry into LAT risk management database by
  06/01/04
• Weekly tracking, updating by ISOC management

21
RFA 3 ISOC Risk Analysis
Number Date Rank Originator Description Mitigation
ISOC-0001 5/15/04 1 B. Craig ISOC lacks accepted architecture and plan for software implementation. Trade study between possible front ends to be completed by 6/15/04. Hires into s/w architecture position.
ISOC-0002 5/15/04 3 B. Craig No response to PDR RFAs Schedule and track RFAs weekly.
ISOC-0003 5/17/04 2 B. Craig Inadequate staffing plan for ISOC. Draft staffing plan in progress, to be released by 06/01 First req issued an offer out to highest priority position.
ISOC-0004 5/21/04 4 B. Craig No facility location identified for ISOC Long-term solution identified, short term space to be requested from SLAC management.
22
RFA 3 ISOC Risk Analysis
Number Date Rank Originator Description Mitigation
ISOC-0005 5/21/04 2 B. Craig No requirements levied on IT and Flt S/W subsystems Mechanism in place with IT, pending with Flt S/W. Implement these only after architecture is defined and accepted.
ISOC-0006 5/21/04 1 B. Craig ISOC will be unable to hold schedule due to staffing delays and unscoped work Definition of work plan follows architecture development. If needed additional support will be requested from LAT management.



23
RFA 7 ISOC Reports

• Specific Request
• Define and document the types of reports that
  will be generated by the ISOC for both internal
  use and for use by external systems (like the MOC
  and GSSC)
• Response
• Reports will be documented in the Operations
  Product ICD (external reports) and LAT Ops Plan
  (internal-only reports)

24
RFA 7 ISOC Reports

• LAT status and planning
• Reported daily (TBR)
• Summary of LAT health status
• Limit violations
• Alerts received
• Current LAT configuration
• Commanding and any other special activities that
  occurred
• Mission planning outlook for near term (time
  period TBD)
• Generated by LOF with automatic and manual inputs
• Published to web server

25
RFA 7 ISOC Reports

• LAT performance
• Reported daily (TBR)
• Quick look science data
• Performance metrics (details TBD)
• Generated by SOG
• Published to web server
• Level 0 data transmission report
• Data transmission metrics (details TBD)
• Automatically generated and sent to MOC following
  receipt of Level 0 data

26
RFA 7 ISOC Reports

• Data Trending
• Housekeeping data
• Environmental data (temp, voltages, currents)
• Derived science quantities
• Trigger efficiency
• Total count rate
• Bright source monitoring
• Includes statistical analysis
• Generated automatically daily/weekly/monthly
• Published to the web

27
RFA 9 - ISOC Lessons Learned

• Issue
• No writeup on lessons learned from visits to
  other instrument/mission operations center
• Resolution
• Members of the ad hoc planning group for the
  definition of the LAT IOC (now ISOC) made visits
  to the operations centers for GP- B (launched
  April, 2004 Stanford Univ., Tom Langenstein
  Brett Stroozas), RHESSI (launched 2002 Berkeley
  Space Sciences Lab., David Smith Manfred
  Bester), and Chandra (launched in 1999 MIT
  Harvard-Smithsonian Center for Astrophysics, Dan
  Schwartz Paul Plucinsky)
• Each of these operations centers integrates
  mission operations with science (instrument)
  operations, and so they are not directly
  comparable to the ISOC in terms of complexity or
  staffing. (The operations center for RHESSI
  includes the ground station.)
• LAT ISOC can learn from others but there are no
  direct models.

28
RFA 9 - Lessons Learned

• The science operations center for GP-B is
  co-located with the science team at Stanford.
  The GP-B data also will be distributed widely to
  collaborating institutions, but the co-location
  at Stanford was deliberate to maximize the
  interaction with the SOC on data issues.
• Colocation important to maximize science.
• The staffing for RHESSI operations is especially
  spare. The facility itself is also used to run
  operations for FAST and CHIPS and the routine
  operations, like scheduling of contacts and
  pipeline processing, are automated. Testbeds
  (simulators for the instrument computers) are
  maintained, and have been found vital for
  understanding anomalies as well as for testing
  flight software updates.
• Testbeds important for flight software updates.

29
RFA 9 - Lessons Learned

• The Chandra Operations Control Center has a room
  with about 4 consoles for the ACIS instrument
  team to monitor and command the instruments. The
  ACIS team has developed an impressive, flexible
  facility for trend analysis. The importance of a
  flexible system that does not require deciding in
  advance what needs to be monitored routinely was
  stressed to us. The ground-based calibration
  data are still actively used, gt4 years into the
  mission. Colocation of the operations (mission
  and instrument) and the ACIS instrument team has
  been important, at least in terms of increased
  efficiency. Instrument team members (like the
  PI) at Penn State can feel out of the loop or
  behind the times.
• Colocation important to keep all science members
  in the loop.

30
RFA 11 LAT Modes

• Specific Request
• LAT Operations Team and Spectrum Astro should
  work together to verify if any interactions
  between LAT modes and spacecraft modes need to
  occur. For example, if a LAT mode change requires
  the spacecraft to change spacecraft mode and/or
  configuration
• Response
• SC modes are understood and accommodate the LAT
  modes as designed

31
RFA 11 LAT Modes, contd
Mission Modes SC Mode LAT Mode
Launch S-Band rcvr/xmit On battery power Off
Early Orbit Inertial capture S-Band rcvr/xmit Sun point with solar arrays tracking Survival
Engineering Inertial point, zenith point, or maneuver Ku-Band xmit, S-Band rcvr/xmit Solar arrays tracking Engineering
Engineering Inertial point, zenith point, or maneuver Ku-Band xmit, S-Band rcvr/xmit Solar arrays tracking Calibration
Engineering Inertial point, zenith point, or maneuver Ku-Band xmit, S-Band rcvr/xmit Solar arrays tracking SAA
Sky Survey Zenith point Ku-Band xmit, S-Band rcvr/xmit Solar arrays tracking Science Mode
Pointed and Repointed Inertial point, maneuver Ku-Band xmit, S-Band rcvr/xmit Solar arrays tracking Science Mode
Safemode Inertial capture, sun point S-Band rcvr/xmit Solar arrays fixed Hardware
Safemode Inertial capture, sun point S-Band rcvr/xmit Solar arrays fixed Survival
Re-Entry Cruise, delta-V S-Band rcvr/xmit Solar arrays tracking Off
32
LAT Modes, contd
33
RFA-12 Number of EEPROM Writes

• Specific Request
• Understand the number of writes to EEPROM on LAT
  from all sources
• Reason
• EEPROMs have a limited number of write cycles
  before they become unreliable
• Response
• Not an issue due to use of TrueFlash File System
  overlay (full description is on RFA response,
  available on ISOC web page)

34
RFA 14 - ISOC Data Storage

• Issue
• No agreement with SLAC management on how data
  storage and processing requirements will be
  funded.
• Resolution
• Estimate of processing and data storage
  requirements performed for SAS by R. DuBois.
  Cost determined and built into ISOC outyear
  funding plan and accepted by SLAC Director of
  Research
• Database costs still being evaluated by database
  working group but now expected to be minimal or
  covered completely by SLAC central computing
  services due to small size ( 1Tb) of database.

35
RFA 14 - Monthly Costs
2005
2007
2008
2006
36
RFA 17 Define LOF/SOG Tools

• Specific Request
• The tools needed to run the LOF/SOG need to be
  specified
• Which HK and science parameters will be monitored
  and in what way?
• What actions would be taken based on the results
  seen with these tools?
• How does the ISOC team know from a design
  perspective that the collection of the described
  IT tools will function in the operations
  environment as an integrated system?
• Reason/Comment
• The overall requirements on the ISOC have been
  given
• Detailed plans for which software
  components/libraries such as Python will be used
  were given
• However, lists of which software tools are
  required to achieve the ISOCs requirements are
  needed

37
RFA 17 - Response

• Which HK and science parameters will be monitored
  and in what way?
• HK parameters are defined in LAT-TD-02905
• Routinely monitored science parameters are
  included within the HK data as Low Rate Science
• Use of high rate science data is being developed
  by SVAC and will be further developed by SOG
• Limits and use of HK data for monitoring are TBD
• What actions would be taken based on the results
  seen with these tools?
• Calibration activities are in development in the
  SVAC
• Contingency actions are TBD

38
RFA 17 - Response, contd

• How does the ISOC team know from a design
  perspective that the collection of the described
  IT tools will function in the operations
  environment as an integrated system?
• Development and testing of ISOC tools is in
  conjunction with IT
• Lists of which software tools are required to
  achieve the ISOCs requirements are needed
• The following slides detail the ISOC software
  tools

39
RFA 17 - ISOC Software Tools
ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written
1 Data Transport and Management
1.1 File retrieval, transmission, and management (internet) X SLAC SCS and Fastcopy
1.2 Archive data files X SLAC SCS
1.3 Parse data into database X
1.3.1 Convert housekeeping data into Engineering Units X
1.4 Data integrity checks X
1.5 Science data reconstruction X
1.6 Calibration tracking X
40
RFA 17 - ISOC Software Tools, contd
ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written
2 Operations Tools
2.1 Electronic logbook X
2.1.1 Reporting X X
2.1.2 Command history X X
2.2 Database management
2.2.1 Command and telemetry X
2.2.2 Science and calibration X
2.3 Archive management SLAC SCS
41
RFA 17 - ISOC Software Tools, contd
ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written
3 Instrument Health (LOF)
3.1 Real time housekeeping telemetry display X
3.2 Historical data trending display X X
3.3 Data monitoring and alarming systems X X
3.3.1 Autonomous reporting X
4 Instrument Diagnostic Tools
4.1 Diagnostic data display and analysis X
4.2 Memory dump parsing FSW
4.3 Testbed management and operation Elec
42
RFA 17 - ISOC Software Tools, contd
ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written
5 Instrument Performance (SOG)
5.1 Visualization tools X X
5.2 Offline calibration SVAC tools X
5.3 Online calibration X
43
RFA 17 - ISOC Software Tools, contd
ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written
6 LAT Commanding Tools
6.1 Command procedure generation and management X
6.1.1 Instrument file generation FSW
6.1.1.1 File management X
6.1.1.2 File validation and verification X
6.1.1.3 File translation to ITOS Perl script
6.1.2 Telecommand generation X X
6.2 Procedure verification and validation on testbed X
6.3 Procedure transmission tools X
6.3.1 Command wrapper generation (for GSSC) X
6.3.2 Command load transfer to GSSC Fastcopy
44
RFA-18 ISOC Operations automation

• Specific Request
• Specify plans and requirements for automation of
  operations software
• Describe the software design for how the
  automation needs will be met
• Response
• Draft of the plans and requirements has been
  completed
• Software design will commence when ISOC software
  engineer is hired

45
RFA-18 ISOC Operations automation

• Data retrieval from MOC
• OPUS
• Archiving raw data
• Dispatch science data to SOG
• Dispatch housekeeping to LOF
• LOF automated processing
• Housekeeping limit checks, warnings
• Science data raw data quality
• Automated reporting of above (web/paging/email)
• Trending
• Weekly/monthly characterization of data
• Calibration tracking computation
• External agency alert retrieval (i.e., SEC, NIST)

46
Roadmap to CDR

• Primary tasks
• 1) Scenario definition
• Work with FSW and IT for all operational modes
  (BC, LB, SC) July 1
• Detailed early orbit plans (BC,LB) July 15
• 2) Contingency operations analysis
• Define possible actions by subsystem (BC,LB) July
  7
• 3) Draft Instrument Ops Section of Mission Plan
  (LB, SC, who at GSFC?) July 15
• 4) Update requirements documents to reflect
  architecture (SC, LB) July 15

47
CDR Prep Schedule

• July 8th Revisit roadmap
• July 21st Laydown
• July 26th Slides to GSFC
• July 29th Dry Run
• August 4th ISOC Peer Level CDR
• August 18th CDR