ALMA Computing IPT Science Software Status Update

1
ALMA Computing IPTScience Software Status Update

• Gianni Raffi, ESO
• ALMA Science Advisory Committee
• 15-16 September 2006

2
1.07 Computing Summary Schedule
(Data from IPS as _________)
Time Now
Array Software gtgt
Observatory Software gtgt
Computing IPT Management and Science Software
Requirements
High Level Analysis and Software Engineering
Common Software and Archive
Executive, Correlator and Control SW
Schedule, Observation Prep, Offline, and
Telescope Calibration
Software Integration and Test
3
Requirements Management

• Goal
• Evaluate development progress with respect to
  requirements
• Provide reporting.
• gt Done via Central database (DOORS) with
  requirements derived from all sources (science,
  technical, operational) that allows us to
• Manage requirements (addition, editing, deletion,
  prioritization, review)
• Assess the status of software development
  (reporting)
• Analyze the impact of requirements changes
• Coordinate requirement tracking for all computing
  subsystems.
• Complete functional team work and release
  development with integration tests (by ITS team)
• Validate implementation with user tests (by SSR
  subsystem scientists) following each release.

4
SSR Testing of Science Software

• Extracted from D.Shepherds page on TWiki at
• http//almasw.hq.eso.org/almasw/bin/view/Usertests
  /WebHomeTest_Plans
• Requirements available on SSR TWiki (unless
  explicitly mentioned below)
• All functional subsystems shown for completeness
  (except ACS framework and ObOps)
• EXEC
• R3.1 - Executive internal pre-test complete.
• R4.0 - user test scheduled to begin October 20.
• Scheduler
• R3.1 - Optical pointing test completed in March
  2006
• R4.0 - Testing planned for early fall 2006. This
  will test the scheduling simulator, particularly
  with weather simulation.
• TelCal
• R3.1 - Features developed during this release
  tested August 21, 2006. Optical pointing
  reduction tests completed March 2006.
• R4.0 holography reduction tests scheduled late
  2006.
• Archive
• R3.1 Basic archive function tested during
  optical pointing integrated user test in March
  2006.
• R4.0 Tests of archive queries (projects,
  monitor point data) scheduled for integrated user
  test late 2006
• Requirements Use Case development for Science
  Archive continues. Granular requirements for
  Archive starting to be produced. ALMA Science
  Archive (ASA) requirements about ready.

5
SSR Testing Status (cont.)

• Correlator
• R3.0 - Testing planned for April, June, and July
  slipped to October 2006.
• Pipeline
• R3.1 - End to end user test (from flagging to
  imaging) is planned for the pipeline in Fall
  2006.
• Quick Look R3.0 - Operations test should be
  possible. It is not yet planned.
• ObsPrep
• R3.1 User test about to conclude.
• Control
• R3.1 Optical pointing features associated
  tested during integrated user test in March 2006.
• R4.0 Holography features scheduled to be tested
  late 2006 during integrated user test.
• Requirements adopting EVLA control requirements
  (modified for ALMA) for ALMA control granular
  requirements. Passed System Requirements Review.
• Offline
• User Test 4 R2.1 - Single baseline data
  reduction and analysis - supporting the ATF. Test
  complete, report written.
• User Focus Group completed March 2006, provided
  input on python user interface.
• User Test 5 Will test single dish reduction
  imaging. Scheduled for Dec 2006 date dependent
  on ACA requirements.
• Note Control and correlator are exercised not
  only at ATF, but also every day in the Prototype
  System Integration (PSI) lab.

6
Function Based Team (FBT) development

• Works to functionality across subsystems (2
  months)
• Based on Technical Use Cases
• Interface issues clarified soon
• Allows more frequent integration
• Some parallelism possible

In principle planning of R4.0 FBTs
7
Latest Developments

• Concluded R3.1 release and integrated in April
• Includes Optical Pointing e2e functionality
  (tested in March at ATF)
• Installed and tested on both prototype antennas
• Installed also in the Prototype System
  Integration Laboratory in Socorro
• Included integrated regression tests
• FBTs so far
• Optical Pointing (completed in March)
• Shared Simulation of Single Field Interferometry
  (SFI) (April)
• Monitor DB optimization (July)
• On-going FBTs
• ALMA Project Data Model (APDM) re-factoring (Aug)
• Validation almost complete. Currently on hold
  behind higher priority Holography work.
• Holography (Aug). Hw for tests in the lab early
  Sept., at ATF late Sept.
• Single Field Interferometry II (to be used at ATF
  just after first fringes- achieved with PSI
  scripts)

8
OPT User Test (mar.06) (1st integrated user
test of ALMA software)

• evaluated by members of the Computing IPT (CIPT)
  in March 2006.
• primary software limitations were found to be
  jlog instability and slowness, lack of coherent
  logging levels across subsystems, archive access
  (query mechanism not documented at the level of
  end users), lack of access to archived monitor
  points, missing CCL offset commands, and uncaught
  exceptions (e.g. Control, TelCal) that caused
  software crashes.
• Many of the interfaces and system responses have
  been designed for the developer.
• Approximately 170 defects, issues, and
  enhancements were identified in this report.
  Roughly 40 (70) of those are defects (14 (8)
  priority 1 and 56 (32) lower priority).

9
Next Developments ()

• R4.1 ( up to June 2007)
• Single Dish (SD) Function Based Team (Feb.) (w/o
  nutator)
• TP mode 1 OTF mapping in continuum Total Power
• TM mode 2 radio pointing measurements using TP
• Test SFI with h/w at ATF (completion of SFI II
  work)
• Readiness Review (external) by subsystem (May)
• R5.0 ( up to Dec 2007)
• SD Nutator, Spectral line (prepare for equipment
  in Chile)
• Interferometry WVR, fast switching, Pointed
  mosaics
• Respond to AIV use of software in Chile
• PAR (Dec.07) Delivery of modes developed thus
  far to Chile
• During all of 2007 support in Chile for antenna
• Acceptance and integration
• System characterization using continuum TP
• () Development plan will be adapted to
  availability of h/w         CIPT formal user
  test will occur twice next year

10
Archive Software

• Persistency support for raw data, pipeline
  results (images and logs), SBs and all other User
  input, site characteristics, monitor data,
  calibration data, catalogs, and all other
  auxiliary information required for ALMA to
  observe.
• Quick look results are temporarily stored in
  the Archive.
• Items not specific to ALMA Science Archive
• User Repository Manage database of registered
  users, support proprietary periods.
• User query, retrieval and monitoring capability.
  (event data R4.0)
• Data replication between OSF, Santiago, ARCs
  (both via network and physical media).
• Use of NGAS supported data replication. Requires
  ALMA testing.
• Request Handler Manage packaging, request
  tracing and delivery of data products to the
  scientific user community (implementation planned
  for Q4 2007)
• High-availability system with failover (test
  implementation Q3 2007)
• Provision for safe backup of all data when more
  than one Archive is operational this may be
  accomplished via data mirroring between Archives.
• Support 6 MB/s average and 60 MB/s peak
  (sustainable for many hours at a time) data rates.

11
Archive Replication between SOC and ARCs Full
description on Archive Twiki at http//almasw.hq.
eso.org/almasw/bin/view/Archive/DatabaseReplicatio
n

• Bulk data replication taken care of by NGAS.
• 2. Transaction based replication for Oracle DB
  OSF-SCO
• 3. Standard DB incremental backups sent via NGAS.

12
ALMA Science Archive (ASA)

• Scientific view on ALMA observations through ALMA
  data products (cubes).
• Preview image, possibly via an interface to
  Science Pipeline.
• Science Archive Requirements document was drafted
  and has been discussed by the SSR.
• Many use cases for the ASA have been drafted and
  were given to the developers. (http//almasw.hq.es
  o.org/almasw/bin/view/Archive/AsaUseCasesList)
• Support for storage and association of advanced
  products.
• Catalogue Support offer access to catalogues
  (calibrators, spectral lines) and calibration
  database (R4.0)
• Consists of two major parts
• ASA portal Main access point to all query and
  retrieval services.
• ASA data repositories and ASA services.
• VO aware by sharing technologies and standards.
• ASA VO registry (planned)
• ASA VO SkyNode (planned)

13
ALMA Science Archive (ASA)

• Exercise use cases for each user type
• ALMA staff, Archive Users/PI and Reduction
  Pipeline
• Test interfaces (query structure, ease of use)
  transfer content (connections between APDM
  structures in ASA made distributed correctly)
  performance.
• Test data transfer from Operations Archive and
  ASA.

Operations Archive (Santiago)
Dataset
ALMA Science Archive
Queries
Query Responses
Commands
Archive Subsystem State
Data
Correlator Data
Processed Data
Quality Data
Archive Users, PI, VO, Staff
ALMA Staff
Reduction Pipeline
14
Add a user to the repository
Support for ASA users Staff PIs User
repository development manage database of
registered users
Manager view of users
User edits his/her contact information
15
Archive table browser prototype
list of ALMA Science Data Model (ASDM) tables
entities table populated by   Correlator (actual
contents not shown)
16
Observing Preparation

• Support creation of Phase 1 proposals and Phase 2
  programs, with submission through the ALMA
  archive.
• APDM refactored (R4)
• Visual Spatial Editor first version. Visual
  Spectral Editor also exists. Final versions of
  both for R6.
• Support for Directors time (R5)
• Special SBs for fixed period scheduling now
  available. (R4)
• Phase I bureaucracy and textual cases (R5).
• Phase I II transition (creation of Phase II
  from approved Phase I) (R5)
• Support all ALMA Observing modes
• So far optical pointing and holography
  observatory modes, and Single Field
  Interferometry (SFI) observing mode.
• infrastructure in place to allow rapid
  development for further modes, R5-6 and on.
• Provide a mode which operates in terms of science
  goals with automated generation of SBs.
• Science goal-based input for Phases I and II
  First wizard now available for SFI. Will
  expand as modes are developed.
• A generic framework for science input collection
  and SB generation is in place.
• Alternative science goal presentation
  (workbooks - more flexible for users than a
  wizard) under development R4.
• Almost complete infrastructure will allow science
  goal input and SB generation to be extended to
  more observing modes. R5-6 and on.

17
Observing Preparation (cont.)

• Exposure time and data rate calculators shall be
  provided.
• ETC complete in maintenance. Supports all
  arrays.
• Data rate estimator first version available,
  requires more user friendly presentation (R5.1)
• Execution likelihood estimator (R5)
• Sensible default values and basic validity
  checking will be provided.
• Infrastructure to support validation and setting
  of (sensible) defaults is in place.
• Needs to be rolled out for all items (R4.1)
• Provide an expert mode with access to technical
  parameters of interest to observers (e.g.,
  detailed control of correlator configuration).
• Almost complete for main array in maintenance
  as other specs emerge. Still details to be
  determined for ACA/TP arrays. (R5)
• Local disk persistency (provided by
  MicroArchive). In maintenance.
• Source list and catalog (target and spectral)
  support
• Infrastructure in place with interim catalogs.
• To be moved to Observatory Characteristics (R4)
• Final catalogs in place by agreement with
  Operations (R6?)
• Widely portable software.
• Supported now, will continue.

18
Spatial Visual Editor
19
Observatory Operations Software (ObOps)

• Passed its PDR at CDR4 (June 2006). Still at
  .5 FTE/yr effort
• Infrastructure Ops Support Database, Data Access
  Module, etc. (R4-R9)
• Shift Log Tool record all day- and night-time
  observing activities
• core features (R4)
• basic features (R5)
• full implementation (R6)
• Observatory Characteristics Database GUI (R4.1)
• Phase One Manager support for proposal review
  process (R6)
• ProjectTracker control and monitor operations
  information flow (R6)
• User Portal integrated access to operations info
• internal (ALMA only) access (R7)
• public access (R9)
• QualityTool data quality assurance (R7)
• DataPacker
• raw data delivery (R7)
• data package production and delivery (R9)
• PublicationsDatabase (R9)

20
Offline software

• Priority 1 requirements. In particular for the
  next release
• Single dish analysis, High dynamic range imaging
  (R4.0)
• Standard calibration (R4.0)
• CASA framework doc.
• Portable to a range of Unix-like operating
  systems.
• Efficient enough in operations to satisfy user
  expectations and for use with the Science
  Pipeline.
• GUI ,CLI, and script interfaces (Image viewer
  data visualization GUIs available)
• Readily installed by end users for processing
  execution. GUI tools for raw data and image
  inspection, flagging, arbitrary plotting.
• Filler for ALMA Export Data Format for
  Interferometric data (single dish ASDM filler to
  be developed when ASDM format defined).
• ASDM extensions, EDF doc (R4.0)
• Extensive support for multi-channel,
  multi-polarization, multi-spectral window,
  multi-pointing flagging, calibration, imaging,
  and analysis for interferometric, single dish,
  and combined data from existing telescopes.
• Simulation capability.
• Processing history, interactive warnings, and
  re-execution facilities.
• Direct access to the ALMA Archive and the VO.
• Atmospheric modeling for the ALMA bands
• Extensive user documentation

21
CASA

• AIPS
• Glish interface unknown, unsupported outside of
  NRAO
• Tasking system based on Glish
• GUI system based on Glish/Tk limited widgets,
  not robust!
• Difficult for external developers to contribute
• Multi-CD binary distribution
• Large monolithic libraries with cross
  dependencies
• No namespace
• Freeze 2006.75

• CASA
• Python interface (community standard) IPython
  extensions
• Direct binding to Python ACS, other frameworks
  readily possible
• Hierarchical set of small libraries with clearly
  defined dependencies
• Namespace protection for integration with other
  code
• RPM distribution mechanism auto-updates possible
• Robust
• Inherits all application code improvements in
  robustness and performance.

22
DataCapture
Maps telescope, observing information into Alma
Science Data Model (ASDM) for post-processing in
near-real time.
23
CASA Timeline

• EVLA external science testing wide band
  calibration imaging (CASA)
• ALMA P1 SSRs complete
• 2008
• EVLA external science testing RFI (CASA)
• NRAO user support (CASA)
• ALMA commissioning support
• EVLA P1 SSRs complete
• 2009-2010
• ALMA/EVLA commissioning support
• Community CASA released/distributed for early
  ALMA/EVLA science
• 2011
• Community CASA released/distributed full user
  support (ALMA/EVLA P1/P2s)

• 2006
• NRAO NAUG testing (AIPS)
• Community Project tests
• ALMA external science testing single-baseline
  commissioning test) (CASA)
• ALMA external science testing single dish
  reduction (CASA)
• ALMA commissioning support (CASA)
• ALMA/EVLA user interface review (CASA)
• EVLA external science testing full
  polarization imaging antenna pointing
  calibration
• 2006.5 AIPS frozen
• 2007
• NRAO User support (CASA)
• ALMA commissioning support (CASA)

24
Offline User Test 4 (report Apr.06)

• Test delayed by SSR (not developers) so the first
  framework changes and python commands could be
  evaluated in a realistic environment of single
  baseline interferometry.
• CASA is a python binding of the C functionality
  that was in AIPS.
• This initial version of CASA had python commands
  that had a one-to-one correspondence with AIPS
  Glish commands.
• first test of the CASA infrastructure and python
  interface.
• focus was to exercise and verify functionality
  for
• editing, visualization, data reduction and
  analysis for single baseline interferometry
• capabilities that are needed at the ALMA Test
  Facility and for commissioning support.
• Testers with extensive experience commissioning
  radio telescopes and/or operating telescopes.
• In general, single-baseline data reduction
  procedures worked with good stability and
  accuracy.
• Additional visualization capabilities to support
  commissioning activities still needed.

25
Offline Development Test Schedule
Community use, command line python
Limited release, mostly command line, some GUIs
Single dish reduction imaging
Python tasks, new framework
Delay late arrival of ACA requirements.
User Focus group
Tools ? Tasks
Single-BL Commissioning at ATF
1st Python interface, new framework
Delay SSR decision to allow testing of 1st
python interface.
Interferometer/TP combo
Complete
VLAGBT BIMA12m
Small Multi-field Mosaic
VLA BIMA mosaics
User Acceptance Tests
Single Field
VLA PdBI 13mm 1/3mm
2003
2004
2005
2006
2007
26
Offline Key Planned Development for 2006/2007

• Commissioning Plan shows heavy use of the offline
  package.
• Change development priorities/details (e.g.
  single baseline analysis software must be ready
  earlier than expected more plotting capabilities
  needed).
• Completed commissioning support test in February
  2006 (1st step before using ATF)
• Single-dish reduction imaging will be based
  on ASAP package
• Single-dish user test scheduled for late 2006
  (dependent on ACA requirements).
• Also evaluate task-based reduction imaging for
  interferometric data.
• Complete ALMA interferometric ASDM filler and new
  viewer (more robust).
• Formal tests have been limited to specific
  datasets to verify development along a defined
  path. This has been necessary for development but
  need to expand to more open use once python
  command line interface is stable
• 2007 2009 Staged plan to release CASA to
  astronomers for scientific use.
• Not final yet, however, key components are

• Nov 2006 CASA user point person starts (guide
  user interface development, documentation help
  new users)
• 2007-2008 Increase number of NRAO scientists a
  few external scientists reduce own data
• Mid 2007 Early release of CASA simulator
• Mid-2008 Release to the public.

27
Science Pipeline Goals

• Automated flagging, calibration and imaging
  with associated quality information for common
  ALMA observing modes. Pipeline processing shall
  be data driven).
• Main purpose is to produce calibrated,
  scientifically useful images.
• Following information will be calculated or
  available for use in pipeline processing
  sideband ratios, temperature scale from an
  atmospheric model, bandpass calibration,
  instrumental polarization, phase rms, antenna
  efficiencies from source with known flux, and
  time variable amplitude and phase curves.
  (heuristics computing bandpass, gain, and flux
  calibrations for single field interferometry
  data, (R3.1-gt)
• Choose whether or not to use WVR corrected if
  both are available, to reuse previous
  calibrations if necessary, to select appropriate
  deconvolution algorithm, to do self-calibration
  or not, to do continuum subtraction or not
  (Choices non-existent or hardcoded, (R4.1-gt).
• Automatically assemble all needed information
  from the Archive (i.e., assemble appropriate
  sources from multiple SBs over multiple epochs /
  configurations). (automatic ASDM retrieval for
  observing modes with simple observing unit set
  structures supported, (R2.1-gt)
• User friendly processing summary and processing
  script which can be modified and used in later
  reprocessing. (processing summary logs are
  written by current pipeline heuristics scripts,
  automatic script logging may be available from
  CASAPY, (R4.0-gt)
• Scripting interface to allow operations to modify
  processing heuristics without changes in compiled
  code. (Scripts will be written in Python, (R4.0)
• Manual mode (e.g., for reprocessing particular
  projects). (R5.0-gt)
• First version of pipeline processing scripts.
  (E2E for single field interferometry (R4.0-gt),
  for single dish (R4.1-gt)

28
Status Heuristics

• Single field interferometry (prototype e2e
  heuristics)
• calibrator source baseline based flagging
  heuristics
• bandpass, gain, flux calibration solution
  heuristics
• calibrator source imaging heuristics
• some quality assessment measures
• calibrator source antenna based flagging
  heuristics
• target source flagging and imaging heuristics
• Single dish (baseline fitting heuristics)
• including line detection and masking heuristics
• Software
• general Python classes for flagging,
  calibrating, and imaging SFI data
• general Python classes for baseline fitting (some
  cleanup required)

Heuristics development slower than expected
(based on 2004 planning) Reasons Technical
CASA framework conversion happening at same time
Developmental more difficult to get flagging
heuristics correct than expected Personnel
ramp up to expertise needed to develop
interferometric heuristics slow.
29
Status

• Pipeline Infrastructure
• Quicklook
• supports parallel observing sessions (independent
  subarrays)
• provides clean separation of processing and
  display functions
• working telescope calibration results monitor and
  GUI
• Science
• working heuristics development framework
  (IPython, CASAPY)
• working set of data processing engines (CASA,
  ASAP)
• heuristics not yet connected to infrastructure
• ASDM filler prototype
• separate ALMA / CASA build environments
• Modules
• full set of tool methods available to heuristics
  from CASAPY for
• table, ms, constants, quanta, autoflag,
  calibrator, and imager
• subset of tool methods available to heuristics
  from CASAPY for
• image, measures
• prototype visibility display tool already
  available to heuristics from CASAPY
• msplot

30
Heuristics User Test 3 Browser Snapshots
PdBI Spectral line
From R3 User test conclusions (mar.06) The
quality of the automatic flagging was judged to
be good by the majority of the testers and the
calibration heuristics were found to be a
reasonable first implementation
31
Pipeline - What is ahead for next year ?

• Science
• E2E single field interferometry heuristics
  running in the ALMA environment
• E2E single dish heuristics
• prototype cluster computing environment
• QuickLook
• user friendly telescope calibration monitor and
  display
• real time visibility display using real time
  filler
• some bad data detection heuristics integrated
  with alarm system

32
Dynamic Scheduling

• Software to automatically choose best SB given
  scientific priority, current environmental
  conditions, current and future configurations,
  and the total pool of possible observations
• Complete interactive - and queue - scheduling
  (R4.0) (R4.0).
• Planning mode simulator (incl. weather data and
  quality - R4.0) (R4.0)
• SB interruption and re-execution. (Special SBs
  for fixed-time scheduling - R4.0) (R5-6)
• Manual override of automated choices. (R4.1)
• Mechanisms for operations to change the weight of
  factors within the dynamic scheduler without
  significant programming. (R4.1)
• Scheduler reporting facilities. (GUI for master
  scheduler, Exec plug-in - R4.1) (R4.1)
• Heuristics delivered by CIPT will require
  optimization by operations. (R5)
• Multiple sub-arrays supported (including taking
  antennas off-,on-line - R4.1) (R4.1 )
• ACA projects operating independently (R5-6)
• Medium term scheduling (R5-6)
• Long term scheduling (R7-8)
• Receiver setup optimizations (R7-9)
• Breakpoint facility low priority. (R7-8)
• Accounting of time spent for various purposes
  (source, programs, observatory calibrations).
  (R?)

33
Operator Master Client
Scheduler
34
TelCal Online (and Offline) Calibrations

• Telescope Calibration takes scans (mainly
  special calibration scans) and performs data
  reduction on them.
• Results are used by Control for further
  observations with improved parameters and by the
  Quick Look Pipeline to display and calibrate
  quick look data.
• Some array calibrations are not required to be
  performed on-line (e.g. Holography, pointing
  models).
• Holography (Pointing focus offset, illumination
  pattern, surface err.) (R4.0)
• Atmospheric calibration (Temperature scale) (R4)
• WVR values (R5)
• Sideband ratios (R4)
• Phase calibration (phase rms, calibration curves)
  (R4)
• Amplitude calibration (calibration curve) (R4)
• Bandbass calibration (R5)
• Reference pointing (publish/reduce scan results)
  (done)
• Focus (partly done)
• Delay (publish/reduce delay scans) (R4.0)
• Pointing Model (optical done)
• Baseline measurement (positions from delays)
  (R4.0)
• Skydip
• Interferometry solvers (done)
• Automatic activation (according to intent) (R4.0)