HMI/AIA Joint Science Operations Center (JSOC) Design

1
HMI/AIA Joint Science Operations Center (JSOC)
Design
HMI01207
Phil Scherrer, Stanford University Jim Aloise,
Stanford University Jerry Drake, LMSAL
2
Overview of HMIAIA Joint Science Operations
Center (JSOC)

• Philip Scherrer
• pscherrer_at_solar.stanford.edu
• 650-723-1504

• HMI AIA Institutional Roles
• Scope of JSOC
• Parts of JSOC, SU LMSAL
• JOC Joint Operations Center
• SDP Science Data Processing

3
HMI AIA Institutional Roles
4
JSOC Scope

• The HMI/AIA Joint SOC consists of two parts
• Science Data Processing (SDP) at Stanford and
  LMSAL
• Joint Operations Center (JOC) at LMSAL
• JSOC JOC includes
• HMI and AIA Commanding and Health Monitoring
• HMI and AIA Engineering support as needed
• JSOC SDP includes
• HMI and AIA Telemetry Data capture (from DDS) and
  archive
• HMI and AIA Level-0 processing and archive
• HMI processing through to level-2 with archiving
  of end products
• AIA processing through level-1a with online
  archive at Stanford
• AIA level-2 processing at LMSAL
• Data export of the above and other HMI and AIA
  products as needed
• JSOC does not include tasks such as
• Science analysis beyond level-2 products
• HMI and AIA EPO
• HMI AIA Co-I science support

5
HMI AIA JSOC Architecture
6
HMI/AIA Joint Science Operations Center (JSOC)
Science Data Processing (SDP)

• Jim Aloise, Stanford University

7
JSOC SDP Agenda

• Significant Level 4 requirements
• SDP Architecture
• SDP Network Design Security
• Test approach
• Implementation plan
• Configuration Management and Control
• Trade Studies and Prototyping Efforts since PDR
• Risks Mitigations

8
DMR JSOC SDP Requirements

• 8000.2.4 Science Data Processing, Archiving and
  Distribution
• Each SOC shall provide the necessary facility,
  software, hardware and staff to receive, process,
  archive and distribute the science data generated
  by its instruments.

My Documents\DMR_SOC_Req.ppt
9
JSOC (Stanford Science Data Processing)
Configuration
DDS
SDP
Pipeine Processing
Datacapture System
Switch
Switch
File Server
ethernet
Data Base Server
Analysis Cluster
heartbeat
Passive
Active
Disk arrays
Server
Server
Fibre Channel Switch
etc.
Disks
Tape Robot
Pipeline Processor
Tape Robot
LAN
jim\My Documents\hardware_config3.vsd
10
Telemetry Data Archive

• Telemetry data is archived twice
• The Data Capture System archives tlm files for
  offsite storage
• Archive tapes are shipped to the offsite location
  and verified for reading
• A feedback mechanism will be established to ack
  the SDP that a tape is verified or that another
  copy needs to be sent
• The Data Capture System copies tlm files to the
  Pipeline Processing System
• The Pipeline Processing System archives tlm data
  for local storage and acks the SDP when it is
  successful
• Only when the SDP has received positive acks on
  both archive copies does it inform the Data
  Capture System to include the tlm file in the
  .arc file to the DDS, who is now free to remove
  the file from its tracking logic
• The lev0 data on the Pipeline Processing System
  is also archived

11
JSOC Data Capture System Components
lev0 module
DDS
Operator
Primary Storage Disks
Utility Libraries
GUI Data Capture Interface
FTP Disks
Record Cache
SUMS Storage Unit Management System
DRMS Data Record Management System
Processing History Log
Tape Drive
Database Server
12
JSOC Pipeline Processing System Components
Pipeline processing plan
Pipeline Operator
SUMS Disks
Processing script, mapfile List of pipeline
modules with needed datasets for input, output
PUI Pipeline User Interface
SUMS Storage Unit Management System
DRMS Data Record Management System
SUMS Tape Farm
Processing History Log
Database Server
13
Storage Unit Management System (SUMS) API

• SUM SUM_open(char dbname)
• Start a session with the SUMS
• int SUM_close(SUM sum)
• End a session with the SUMS
• int SUM_alloc(SUM sum)
• Allocate a storage unit on the disks
• int SUM_get(SUM sum)
• Get the requested storage units
• int SUM_put(SUM sum)
• Put a previously allocated storage unit
• int SUM_poll(SUM sum)
• Check if a previous request is complete
• int SUM_wait(SUM sum)
• Wait until previous request is complete

14
Data Record Management System (DRMS) Logical Data
Organization
JSOC Data Series
Data records for series hmi_fd_V
Single hmi_fd_V data record
Keywords RECORDNUM 12345 Unique serial
number SERIESNUM 5531704 Slots since
epoch. T_OBS 2009.01.05_232240_TAI DATAMIN
-2.537730543544E03 DATAMAX
1.935749511719E03 ... P_ANGLE
LINKORBIT,KEYWORDSOLAR_P
hmi_lev0_cam1_fg
hmi_lev1_fd_V12345
aia_lev0_cont1700
hmi_lev1_fd_V12346
hmi_lev1_fd_M
hmi_lev1_fd_V12347
hmi_lev1_fd_V
Links ORBIT hmi_lev0_orbit, SERIESNUM
221268160 CALTABLE hmi_lev0_dopcal, RECORDNUM
7 L1 hmi_lev0_cam1_fg, RECORDNUM 42345232 R1
hmi_lev0_cam1_fg, RECORDNUM 42345233
hmi_lev1_fd_V12348
aia_lev0_FE171
hmi_lev1_fd_V12349

hmi_lev1_fd_V12350
hmi_lev1_fd_V12351
hmi_lev1_fd_V12352
Data Segments Velocity
hmi_lev1_fd_V12353

Storage Unit Directory
15
S/W Configuration Items

• Storage Unit Management System (SUMS)
• Data Record Management System (DRMS)
• Pipeline User Interface (PUI)
• Event Manager
• Science Libs
• Pipeline Utility Libs
• Lev0 Module
• Lev1 Module
• Standard Products Modules
• Export Support

16
H/W Configuration
17
H/W Configuration (cont)
18
JSOC Data Export System
Researcher B
Space Weather
API
Script Access
Keywords Range
Researcher A
Selected Data Records
Drilldown
Overview New/Avail Statistics
General Public
DRMS
VSO Virtual Solar Observatory DRMS Data
Record Mgmt Sys
19
Performance Analysis

• AIA/HMI combined data volume 2 PB/yr 60 MB/s
• read write x 2
• quick look final x 2
• one reprocessing x 2
• 25 duty cycle x 4
• 2 GB/s (disk)
• 0.5 GB/s (tape)
• NFS over gigabit ethernet (50-100MB/s/channel) 1
  4 GB/s
• 4 8 channels per server, 5 servers (today)
• SAIT-1 native transfer rate (25-30 MB/s/drive)
  0.5 0.6 GB/s
• 10 SAIT-1 drives per library, 2 libraries (today)

20
Heritage

• MDI ground system design and implementation
• MDI production processing for 9 years
• MDI sustaining engineering for h/w and s/w
  upgrades
• MDI lessons learned folded into JSOC SDP design

21
Stanford/Lockheed Connections
Stanford
DDS
NASA AMES
LMSAL
1 Gb Private line
MOC
White Net
22
Test Approach

• Telemetry data is simulated with MDI data
  packaged in SDO VCDUs
• Various telemetry pathologies can be created to
  validate telemetry processing
• The DDS is simulated by packaging the VCDUs into
  files and injecting them into the SOC front end
  using the DDS/SOC protocol
• The lev0 processing is validated by comparing
  input and output images
• Timing analysis and throughput studies will be
  performed to evaluate h/w and s/w tradeoffs
• Instrument sunlight tests will be used to
  validate the lev1 processing
• Each s/w library and subsystem has a regression
  test suite that is run to verify each new CM
  release
• Full end to end testing is accomplished via the
  mission IT plan

23
Development Test Plans and Procurement Schedule

• HMI and AIA Data EGSE installed
• Prototype for I/F testing with GS Mar, 2005
  onward
• Version 2 to support flight inst. June 2005
• JSOC Capture System
• Purchase computers Summer 2006
• Support DDS testing Fall 2006
• Final system installed Spring 2007
• JSOC SDP Infrastructure, SUMS, DRMS, PUI
• Prototype testing of core system June 2005
• Fully functional Dec, 2005
• Purchase computers for JSOC Jan, 2007
• Infrastructure Operational March, 2007
• Integration with DDS March-June , 2007
• Data Product Modules Jan, 2008
• During Phase-E
• - Add media and disk farm capacity in staged
  plan, half-year or yearly increments
• - First two years of mission continue Co-I
  pipeline testing support

24
Configuration Management Control

• Capture System
• Managed by JSOC CCB
• Controlled in CVS
• SUMS, DRMS, PUI, etc. Infrastructure
• Managed by JSOC CCB after launch
• Controlled in CVS
• PUI Processing Tables
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Level 0,1 Pipeline Modules
• Managed by HMI and/or AIA Instrument Scientist
• Controlled in CVS
• Science Analysis Pipeline Modules
• Managed by program author
• Controlled in CVS

25
CM with CVS
26
Prototype/Trade Studies Since PDR

• MDI Heritage software for EGSE Science W/S
  (currently running)
• Study to separate keywords from image data in a
  keyword Database
• MDI database optimization study
• MDI internal data set representation study
• Storage management trade study (commercial HSM
  vs. MDI heritage)
• Archive media study to replace MDI Ampex robotic
  units
• Storage Area Network (SAN) / Network Attached
  Storage (NAS) trade study
• Location of offsite media storage

27
Risks and Mitigations

• Physical location of JSOC
• - Tracking through Stanford Management
• - Rent space off-campus
• Cost risk in assumed rate of computer technology
  development
• - Advances already made
• Database Performance
• - Prototyping underway
• - Design alternatives
• - Higher performance equipment

28
HMI/AIA Joint Science Operations Center (JSOC)
Joint Operations Center (JOC)

• Jerry Drake, LMSAL

29
Joint Operations Center (JOC) Agenda

• Significant Level 4 Requirements
• HMI AIA JOC Architecture
• Hardware Configuration Items (HWCIs)
• Software Configuration Items (SWCIs)
• Heritage
• Network Design and Security
• Test Approach
• Implementation Plan/Sustaining Support
• Configuration Management (CM)
• Trade Studies and Prototyping Efforts since PDR
• Risks Mitigations
• Procurement Status
• Integrated Schedule
• Risks Mitigations
• Issues Concerns

30
JOC Requirements Sources

• 464-GS-REQ-0005, Detailed Mission Requirements
  (DMR) for SDO Mission
• 464-SYS-REQ-0004, Mission Requirements Document
• 464-GS-ICD-0001, MOC-SOC ICD
• 464-HMI-ICD-0002, Spacecraft to HMI Interface
  Control Document464-AIA-ICD-0011, Spacecraft to
  AIA Interface Control Document
• NPR 2810, NASA Security Procedures and Guidelines
• 464-GS-PLAN-0041, SDO Flight Operations Plan
  (FOP)
• 464-GS-PLAN-0042, SDO Database Format Control
  Document
• 464-GS-LEGL-0040, Operations Agreement (OA)
  between the SDO MOC and the HMI SOC
• 464-GS-LEGL-0041, Operations Agreement (OA)
  between the SDO MOC and the AIA SOC
• 464-GS-PLAN-0010, Operations Concept Document

31
Significant Level 4 Requirements

• The following significant level 4 requirements
  are from 464-GS-REQ-0005, Detailed Mission
  Requirements (DMR) for SDO Mission
• 8000.2.1, Instrument Nominal Operations
• Each SOC shall provide the necessary facility,
  software, hardware and staff to operate their
  instruments and ensure their health and safety.
• 8000.3, Security 
• Each SOC shall comply with NASA security
  requirements commensurate with a science
  operations facility as documented under NPG
  2810.1 and its successors. This includes
  information technology (IT) security for
  communications with the MOC and DDS, as well as
  physical security at the SOC facilities.
• 8100, SOC Instrument Operations
• Each SOC shall operate their respective
  instruments and be responsible for health and
  safety monitoring, science planning, and command
  generation.
• 8101, Health and Safety Monitoring
• Each SOC shall monitor the health and safety of
  their instruments and be responsive to anomaly
  conditions 24 hours a day, 7 days a week.
• 8103.5, Flight Software Maintenance
• Each SOC is responsible for maintaining the
  flight software resident on their instruments
  on-board processor. Instrument flight software
  updates shall be coordinated with the MOC and the
  other instrument teams.
• 8200, SOC Interface Requirements
• Each SOC shall have two interfaces with the rest
  of the SDO ground system the MOC at GSFC,
  documented in the MOC-to-SOC ICD, and the DDS at
  WSC, documented in the DDS-to-SOC ICD.

32
JSOC Architecture
33
Operations HW Architecture
Dedicated T1/FT1 Lines (redundant pair)
LMSAL (Room in B. 252)
Stanford University
Tlm socket 1
Tlm socket 2
Tlm socket 2
Tlm socket 1
Cmd socket 1
Cmd socket 1
Tlm RT HK PBK HK
Cmds Cmd logs
SPAREOPS1
SPAREOPS2
SDP QL image production
HMIOPS1 (Sun)
AIAOPS1 (Sun)
SPARE EGSE
HMIOPS2 (Sun)
AIAOPS2 (Sun)
HMIOPS3 LMSAL EGSE
HMI LMSAL EGSE
AIA LMSAL EGSE
AIAOPS3 LMSAL EGSE
Tlm RT HK PBK HK
HMIQL QL image display Planning Analysis
Cmd logs
HMI AIA Web Server (PC or Sun)
DDS Routers (WSC)
External network
Cmd logs
DDS OC3 Lines
Tlm RT HK PBK HK
34
LMSAL EGSE Configuration
LMSAL EGSE
LMSAL EGSE
S-band
HK Tlm
CMDS
Dedicated T1 Communications Line (redundant pair)
35
FSW Maintenance HW Configuration
1553
UART
Port 1 38.4 kbaud
Port 2 153.6 kbaud
Ethernet
Ethernet
LMSAL/Stanford EGSE Cmds Tlm Images
STOL Archives
SW Development (UART port)
Sun hmifsw1
36
Hardware Configuration Items

• Operations Hardware
• LMSAL EGSE composed of sets of two Sun
  workstations running Solaris
• Three sets at LMSAL (HMIOPS, AIAOPS and SPAREOPS)
• One set at Stanford (HMIOPS3 and AIAOPS3)
• One set at MOC (GSFC)
• Both connect through sockets to the MOC
• Socket interface implemented
• LMSAL EGSE
• Spacecraft Simulator (SSIM) provided by NASA SDO
  Project
• One Sun workstation (primary) used for commanding
  and real-time telemetry
• The other Sun workstation is used for
• Playback and Level-0 24 hour telemetry from MOC
• Backup for the primary MOC command and real-time
  telemetry connection
• Quick-look analysis of housekeeping data
• Two Suns (for Quick Look image display) and one
  PC (for web server)
• Flight Software (FSW) Maintenance Hardware
• Software development Sun workstation
• LMSAL EGSE Sun workstation
• Spacecraft Simulator (SSIM) provided by NASA

37
LMSAL EGSE Software

• Core code
• Command processor
• Telemetry processor
• Screen displays
• Event handling
• Logging
• Screen displays
• Limit checking
• STOL processor
• STOL (Special Test and Operations Language)
• Experiment Interface Program (EIP)
• Unique to each program at LMSAL
• Contains socket communications protocol
• Instrument-specific command information from
  database used here
• Will add functionality to contact (by page or
  phone) personnel in case of a limit exceedance
  (from the Limit Check module) or certain events
  as identifed by the Event module
• Tiered calling protocol
• Everyone on list called soon (within 30 minutes)
  if no response to pages or phone call drags out

38
Software Configuration Items (1 of 2)

• LMSAL EGSE
• Core code
• Command processor
• Telemetry processor
• Event handling
• Limit checking
• STOL processor
• Experiment Interface Program (EIP)
• Unique to each program at LMSAL
• Instrument-specific command information from
  database used here
• Contains MOC socket communications protocol
• Automatic Notification System contained in EIP
• Flight Software
• SUROM
• KERNEL (OS plus basic cmd tlm architecture)
• HMI FSW
• AIA FSW
• Tornado (WindRiver development system)
• Green Hills Multi development environment

39
Software Configuration Items (2 of 2)

• Spacecraft Simulator (SSIM)
• NTGSE (HSB and 1553 communications)
• ASIST
• Command Telemetry Database
• Database Tools (programs)
• Database Files
• STOL Procedures (run on LMSAL EGSE)
• Ground Utility Tools
• Binary load generation tools convert object
  modules or tables into binary upload format
• Database tools control command and telemetry
  database
• Analysis Tools
• IDL programs
• Trending programs
• Planning Tools
• Software Simulator
• Needed to check out observing sequences before
  use on-orbit

40
Process Management
Supervisor
Daemon, always running
PROC _MGR
EGSE
Spawn once and manage, watchdog on unexpected
process exit
STOL_EXEC
STOL_COMP
CMD_ CON
EVENT _MGR
LIMIT_MON
TM_MGR
Spawn once and manage, watchdog on unexpected
process exit
Spawn (and respawn if needed) and
manage, watchdog on unexpected process exit
Multiple copies started by user
SCREEN
SDO_OPS_TM
SDO_OPS_TC
41
Run-time Database Creation
42
Telemetry Subsystem Data Flow
Shared Memory Partition
Event_mgr
Telemetry Decomutation Table, Analog Polynomial
Conversion Table Digital to String Conversion
Table Limit Set Definition Table
Snapshot File
Limit Events
S/C SIM (NTGSE)
EGSE Events
Snapshot Command
TM_MGR Engr. Conv. Limit Checking
Screen (GUI)
SDO_OPS_TM
CCSDS Telemetry Packets
CCSDS Telemetry Packets
Raw/Engr. TM Data, Limit Status
Raw Telemetry Files
43
Command Subsystem Data Flow
44
Heritage

• Initially used on MDI and still in use for
  orbital operations
• Adopted for TRACE and still in use for orbital
  operations
• Suggest visit to SOHO and TRACE EOFs at GSFC
  (Bldg. 14)
• Upgraded version currently used on Solar-B FPP
  and SXI
• FPP is in instrument IT SXI is at Observatory
  IT
• Will be used on both HMI AIA with the following
  differences due to command and telemetry
  mnemonics
• Databases
• STOL procedures
• Displays
• Software has evolved over more than a a decade as
  an LMSAL resource

45
Code Reuse
Name of code Percent reuse Percent complete
EIP 95 99
Core 90 95
Automatic Notification System yet to be added
46
Network Design and Security

• Computers used for commanding (2, one for HMI and
  one for AIA)
• Only one socket per instrument for commanding
• Only network connection is to dedicated line with
  MOC (air-gapped)
• Operations equipment in room in B. 252
• B. 252 is card-key controlled
• Room containing SDO Operations cypher-lock
  controlled (if necessary)
• Each user must have an individual account
• Operations computers must be
• Access controlled (proper password length)
• Logging
• Assigned System Administrators
• Requirements on all personnel using systems
  connected to the SDO network for commanding
• National Agency Check (NAC) Takes 7 12 months
  to complete
• Requirements on all personnel using systems
  connected to the NASA networks
• Annual NASA security training required (1 2
  hours)
• Requirements currently under review
• NASA security to visit LMSAL and Stanford in the
  next few months to review the HMI and AIA security

47
Test Approach

• Test Approach
• Run LMSAL EGSE Acceptance Test procedure
• Update Acceptance Test Procedure if new
  functionality added or additional unknown
  problems discovered and fixed
• Run with Spacecraft Simulator (SSIM) provided by
  SDO Project
• Run with spacecraft IT system (interfaces are to
  be identical to the maximum extent practical)
• Tests in IT and with MOC as called for in SDO
  Ground System schedule
• Details in Integrated Schedule
• LMSAL EGSE in use
• Used during instrument development for both
  hardware and software
• Brassboard instruments
• Flight instruments
• Spacecraft IT
• On-orbit operation
• LMSAL EGSE same for HMI AIA except
• Command and telemetry database files (different
  mnemonics)
• Instrument specific display pages
• Flight Software Development system
• Currently in use for flight software development
• Five copies will exist (3 currently) 2 for
  brassboard, 2 for flight, 1 for software
  development

48
Implementation Plan/Sustaining Support

• Implementation plan
• Purchase pairs of Sun workstations at the
  appropriate times in the future (start process of
  screening 2.5 months before need date)
• Install current version of LMSAL EGSE and
  databases
• Acceptance Test LMSAL EGSE
• Sustaining Support
• Organizations Providing Support
• LMSAL Department ADBS
• All software maintenance support provided by
  LMSAL personnel
• Flight software and ground tools
• Documented
• Developers available for maintenance if necessary
• Types of Support
• Software
• LMSAL EGSE
• LMSAL EGSE is maintained long-term due to
  multi-mission usage
• Flight Software
• Hardware
• LMSAL EGSE
• Hardware support under warranty or contract

49
Deliverables HW/SW/Documentation

• Hardware
• LMSAL EGSE (Sun computers)
• PC
• Software
• LMSAL EGSE
• Documentation
• LMSAL EGSE Users Guide DEP0304
• LMSAL EGSE SDO Experiment Interface Users
  Guide HMI01131
• HMI Flight Software Users Guide 2H00782
• AIA Flight Software Users Guide 2T00175
• Users Guide for each ground tool

50
Configuration Management (CM)

• Flight Software
• Managed by HMI and/or AIA CCB respectively
• Controlled in CVS (Concurrent Version System)
• LMSAL EGSE
• Managed by HMI and/or AIA CCB and LMSAL
  departmental CCB respectively
• Controlled in SCCS (Source Code Control System)
• Ground Tools
• Managed by HMI and/or AIA CCB respectively
• Controlled in CVS
• Database
• Managed by HMI and/or AIA CCB respectively
• Controlled in CVS
• Version number in telemetry
• Tables
• Managed by HMI and/or AIA CCB respectively
• Controlled in CVS
• Keep track of flight software configuration
• Directory dumps of RAM and EEPROM provide
  verification of currently loaded software

51
Trade Studies Prototyping Efforts Since PDR

• No trade studies related to JOC
• Prototyping consists of deploying operational
  LMSAL EGSE and flight software systems
• Previous presentations (reviews)
• Mission Operations Peer Review 2004-02-05 HMI0056
  0
• SDO Ground Systems PDR
• Instrument Operations Concept 2004-04-21 HMI0064
  3
• Telemetry Control Design 2004-04-22 HMI00644
• HMI/AIA SOC Design Walkthrough 2004-09-09
• EGSE Limit Checking
• Automated Notification System
• HMI CDR 2004-11-15 AIA01037
• AIA CDR 2005-02-16, 17 HMI00928
• JSOC Peer Review 2005-03-17 HMI01086
• No changes to report (other than continued
  lower-level detailed development)

52
Procurement Status

• Procurement Schedule
• Purchase Suns for flight LMSAL EGSE systems June
  2005
• Purchase Suns for MOC testing (SPAREOPS1
  2) July 2006
• Purchase Sun workstations for flight
• HMIOPS1 2 October 2006
• AIAOPS1 2 October 2006
• Purchase Suns PC for Quick Look/web page in
  JOC October 2006

53
Integrated Schedule

• Spacecraft Simulators (SSIM)
• HMI AIA SSIMs 1 delivered
• HMI SSIM 2 July 2005
• AIA SSIM 2 August 2005
• Develop Automatic Notification System November
  2005
• Real-Time Low-Rate Data Line installation December
  2005
• HMI
• October 2006
• HK Telemetry Telemetry Database
• November 2006
• HMI Commanding Command Database
• HMI FDS Products
• HMI Mission Support Products
• HMI Special Procedures (Definition)
• AIA
• November 2006
• HK Telemetry Telemetry Database
• December 2006
• HMI Commanding Command Database

54
Risks and Mitigations/Issues Concerns

• No known risks
• No issues or concerns