RadioNet FP7 planning session Proposals for JRAs on algorithm and software development

1
RadioNet FP7 planning sessionProposals for
JRAson algorithm and software development

• Huib Jan van Langevelde, JIVE

2
Input to this talk

• 7 became 8, reduced to 6 contributions

MAGIX
3
Some general remarks

• Embarrassed to present project proposals for
  others
• Trying to do these justice
• These project proposals show
• It is the interface to the user and his/her
  science
• And it is still not up to speed with the
  instruments available
• But generally speaking these JRAs are modest
  size
• Compared to the investments in AIPS, AIPS,
  CASA, Miriad
• And they force distributed effort, often with
  temporary staff
• Having done ALBUS in FP6 has some consequences
• Some of the proposed topics are not so new
• Even though we cannot claim everything is solved
• Like to keep some of the efforts alive
• Like to keep some of the people in the system
• We have only just started

Software development deserves a place in RadioNet
FP7
4
ALBUS Overview

• Part 2 Imaging
• Parallelization (ASTRON)
• Bottlenecks
• MIRIAD scripting
• Wide Band Imaging (JBO)
• Analysis
• Implement
• Test
• Wide field Imaging (JIVE)
• Analysis
• Implement
• New 3 Software infrastructure
• ParselTongue
• Scripting
• Calibration access
• Distributed computing

• Part 1 Enhancing the product
• Calibration Transfer (JIVE)
• Antenna Gains
• Phase Cal Tones
• Ionospheric calibration (JIVE)
• Global GPS
• Global model
• Tropospheric calibration (MPI)
• Using GPS
• WVR
• Frequency
• Post correlator processing (JIVE)
• PCInt data flow
• Target selection
• Web portal

90, report needed
D O N E
70, evaluating tests
20, simulations
50, starting GPS
5, requirements
70, gone public
5, requirements
5
What we have learned from ALBUS

• We can do software projects
• Even when they are necessarily distributed
  between partners
• They can be fun, despite the boundary conditions
• Some (few) of the rules enforce some
  professionalism
• There is enough potential in RadioNet partners
• Software can reach the user community
  (eventually)
• But for success I think it is important that
• Overall aim is realistic and concrete
• We take incremental steps to reach intermediate
  functionality
• Workpackages match local ambitions and are
  self-contained
• EC funded positions come in 2 or 3 manyear
  chunks, not smaller
• Possible exception are FC partners
• Have a strong link to scientists with a direct
  application in mind
• There is a PI that has time for the management
  overhead

6

• CARTHAGE
• Cooperation on Advanced Radio-interferometry
  TecHniques for Astrometry and Geodesy in Europe
• P. Charlot
• Bordeaux Observatory

• Objectives to develop algorithms and observing
  strategies for accurate astrometry and
  geodesy with future radio-interferometers
• Instruments to be considered SKA, IVS-2010,
  e-MERLIN, VSOP-2
• New capabilities to be studied
• multi-beaming
• wide-band group delays
• orbiting telescope

7

• WP1 Multi-beaming

• WP1a creating an inertial celestial frame with
  a multi-beam instrument
• Simultaneous observing of widely-separated
  sources allows one to build a coordinate system
  based on the arc-length between sources.
• Celestial frame then does not depend on Earth
  orientation (it cancels in the analysis,
  Dravskikh et al. 1975, Arias 1990) ? improved
  accuracy
• Application possible with future
  radio-interferometers
• IVS-2010 network (which may have 2-3 antennas per
  site)
• SKA (which may have 10s of beams)
• WP1b phase-referenced astrometry with a
  multi-beam instrument
• Cluster-cluster test observations made with
  WSRT-VLA-MERLIN baselines (Rioja et al. 1997,
  Porcas et al. 2003).
• Dual-beam phase-referenced observations now
  underway with VERA.
• Analysis strategies to be developed for complex
  networks with many beams like SKA
• Goals study algorithms, develop simulation
  tools, acquire test data and assess astrometric
  accuracy for both wide-angle (WP1a) and
  narrow-angle (WP1b) observing conditions.

8

• WP2 Wide-band group delays

• e-MERLIN will be able to record 4 GHz bandwidth,
  hence permitting to construct very-accurate group
  delays in the geodetic way.
• Such group delays may be used for
• measuring the astrometric positions of thousands
  of weak sources
• monitoring the geodetic positions of the e-MERLIN
  telescopes
• Goals develop modeling for wide-band group
  delays, acquire test data, and assess geodetic
  and astrometric accuracy

WP3 Orbiting VLBI telescopes

• VSOP-2 will provide the opportunity to measure
  ground-space VLBI delays which may be used for
  orbit reconstruction if these are accurate
  enough.
• Software does exist at CNES Toulouse to analyse
  such space-VLBI data.
• Successful demonstration with HALCA satellite
  (Meyer et al. 2000).
• Goals adapt existing software to simulate
  VSOP-2 delay observations, carry out simulations
  for various conditions of the VSOP-2 satellite,
  assess accuracy of orbit reconstruction based on
  VLBI data

9

• Timeline, partners and manpower

• Timeline and deliverables
• Year 0 -------------- 1 -------------- 2
  -------------- 3 -------------- 4 --------------
  5
• WP1 Algorithms Simulation tools
  Test data Conclude¹
• WP2 Modeling Test data
  Conclude¹
• WP3 Software Simulations
  Conclude¹
• ¹ Assess astrometric/geodetic accuracy,
  recommend observing analysis strategies and
  deliver corresponding reports for the 4
  instruments (SKA, IVS-2010, e-MERLIN, VSOP-2)
• Potential partners
• WP1 Bordeaux, MPIfR, OAN, Uni-Bonn, Paris Obs.,
  IVS groups,...
• WP2 UMAN, Bordeaux, MPIfR,...
• WP3 Bordeaux, SGO, CNES Toulouse, JIVE, VSOP-2
  team...
• Tentative budget
• WP1 4 FTE per year over 5 years
• WP2 2 FTE/year over 3 years
• WP3 2 FTE/year over 3 years

10
MAGIX(Modeling and Analysis Generic Interface to
eXternal numerical codes)

• D.Muders, P. Schilke, F. Wyrowski, H. Hafok
  (MPIfR)
• F. Boone, M.L. Dubernet (Observatoire de Paris,
  LERMA)
• U. Lang (Cologne University)
• E. v. Dishoeck, M. Hogerheijde (Leiden University)

11
Problem

• Analysis and Modeling of current and future (e.g.
  ALMA) large astronomical data sets is essential
• This task is currently quite difficult because
• The fitting and simulation tools often use
  proprietary non-standardized interfaces
• Iterations through the multi-dimensional
  parameter space must be performed manually

12
Proposed Solution

• Develop a core framework of generic interfaces to
  model parameters, model outputs and astronomical
  data sets
• Add a generic engine to allow iterating selected
  model parameters automatically based on
  minimization algorithms that compare model output
  and observed data
• The effort to add new numerical codes to the
  framework is thus minimized

13
DALIA (Direct Approach to spectral Line Analysis)
Prototype
14
FP7 Deliverables

• Plan
• Generic interfaces (2006)
• Iterating engine (2006-2007)
• Sample applications (2007-2008)
• FP7 FTEs
• 2-3 software engineers
• 1 analysis oriented scientist

15
Archiving For Radio TelescopesWidening Access
to Radio Data

• Gary Fuller, Univ of Manchester

16
Archiving For Radio TelescopesWidening Access
to Radio Data

• Goal make radio data more easily accessible to
  all (especially through Virtual Observatories)
• Widening access, strengthening trans-observatory
  science and so increasing science productivity
• Ultimately want end-to-end, proposal-to-publicati
  on, system
• but will concentrate on data product end of
  system
• Many issues common to different (radio)
  telescopes /observatories suggests the need for a
  common approach
• Experience in Manchester Three PDRA's working
  on design and implementation of ALMA Archive and
  ALMA Science Archive.
• Current interest from Manchester, JIVE, ESO (ALMA
  archive lead ESO archive lead), Nançay, Oxford,
  IRAM (?).
• Joint Research Activity

17
Proposed Project Tasks

• Usecase development
• in particular projects which access multiple
  radio, especially interferometer, archives and
  multi-dimensional datasets.
• Software technology development. Examples include
• explore use of ALMA Science Data Model (ASDM) as
  a science metadata system for other radio
  telescopes
• user request processing
• data and metadata storage and access methods and
  technology
• cube and image generation stored vs. on-the-fly
• user interface
• Work with IVOA (and partners) developing data
  and interoperability standards
• Implementation
• User tests and demonstrations

18
Focal plane array calibration and imaging software
Focal Plane Array Calibration and
Imaging Parallel Processing Software
Carole Jackson Tim Cornwell, ATNF
19
Focal plane array calibration and imaging software

• xNTD requires software for calibration and
  imaging using FPAs on synthesis radio telescopes
• Many new challenges, both in software and
  algorithms
• e.g. element gains, dissimilar beams, field
  rotation, polarization
• Platform Probably CASA libraries python
• Existing collaborations KAT being discussed
• Resources Estimate 10 FTE years over 4 years
• Deliverables
• Software for storing, editing, calibrating and
  imaging FPA data
• Necessary algorithms for dealing with calibration
  and imaging challenges specific to FPAs

20
Parallel processing software for FPA and AA
synthesis telescopes

• xNTD requires about 20Tflops processing
• Mostly embarrassingly parallel (spectral
  channels)
• Some non EP (continuum)
• Platform CASA libraries LOFAR middleware
• Collaborations With KAT and LOFAR (in
  discussion)
• Resources about 5-10 FTEs over 4 years
• Deliverables
• Calibration and imaging software for FPA-based
  telescopes capable of running on very large
  clusters
• Evaluate inner loops adapted to special purpose
  hardware

21
Possibilities for collaboration

• NTD will start producing data and insight
  relevant to FPA processing later this year
• In addition, NTD has an ongoing development
  effort in novel FPA technology
• We would welcome collaboration on the software
  issues mentioned before
• Knowledge and experience in HPC is particularly
  helpful
• We have guaranteed and also likely access to
  substantial clusters and other computers suitable
  for testing

22
Wide field Imaging and the Analysis of Extremely
Large Data Sets

• Notes from Paul Alexander, Steve Rawlings
• Cambridge Oxford, UK

23
Wide-field imaging

• For the SKA to work we need dynamic ranges of 107
  1 how do we achieve these?
• Need to develop improved wide-field imaging
  algorithms considering for example
• Variations in instrumental/atmospheric phase
  across the field-of-view
• The effects of polarization purity and variations
  across the field-of-view
• Dealing with (polarized) sources in the
  side-lobes rather than the main beam
• Dealing with the effects of antenna pointing and
  deformation errors
• The effects of very wide-band effects
• Differential ionospheric and tropospheric effects

24
Grid-enabled Parallel Algorithmsand Large Data
Sets

• We need to develop grid-enabled parallel
  algorithms to deal with the extremely large
  datasets we will have from the next generation of
  instruments
• Challenge is that some aspects of our current
  algorithms are data-flow limited (e.g. going from
  sky- to instrument-model)
• Now may be a good time to look again at the whole
  data analysis model for interferometric data
• What is the raw data? For a fully digital
  telescope perhaps it should be considered the
  digitised antenna outputs and NOT the partially
  calibrated visibilities
• Develop from first principles new
  algorithms/methods working on these raw antenna
  data which are from the outset grid-enabled

25
Grid-enabled Parallel Algorithmsand Large Data
Sets

• Other implications of extremely large data sets
• Need to consider the complete data flow from
  antenna to end-user
• The model of giving the end-user partially
  calibrated visibilities will not work for the SKA
  for example
• The output must be what is needed
• Need to build feature extraction (e.g. find a
  galaxy in position-velocity space) into the
  complete pipeline
• Learn from other areas space-CMB experiments,
  particle physics, gravitational wave communities

26
(No Transcript)
27
User perspective, classic users

• Starting point
• Control the loop gain on their clean run
• Do some special tricks or non-standard processing
• Or for teaching and instruction, train new
  experts
• Data from classical radio instruments
• Upgraded considerably (eVLBI, eMERLIN, eVLA)
• New telescopes with new software tools
• And different, pipelined data products, often
  calibrated and imaged
• ALMA, LOFAR, SKA pathfinders
• Two scenarios
• use the new software for traditional cm data?
• or the old software to access the new
  instruments?

There will be radio-astronomers in 2010 who want
to take control over calibration imaging
28
Things we know for sure

• Some people still want to use AIPS (or Miriad)
• Because they are conservative scientists
• Who trust the embedded algorithms
• Or are just busy and not committed to learn
  something new
• AIPS cannot survive with current data access
  methods
• Python is the interface for more than one package
• We like to continue ParselTongue support (AIPS,
  maybe Miriad)
• We hope to have done some high level
  parallelization
• One could develop a high level user interface
• ALMA ( eVLA) will be using casa
• And LOFAR software is based on same environment
• Hardware capabilities will scale up (SKA)
• Improves to processing capacity
• As well as the data volumes
• I believe there is no limit to what people would
  like to do if the resources were available

29
Where we want to go
Port existing algorithms to the casa
era Preparing the cm user community for SKA

• Port traditional algorithms to casa/LOFAR
  environment
• Focus on global fringe fitting, important for all
  long baselines
• And improve the algorithm if possible
• This is a way to continue support for recent (or
  upcoming) ALBUS algorithms
• Interested to extend wide field to mosaicing and
  maybe focal plane arrays
• Build on the ParselTongue layer
• Run various packages from one interface
• Extend the user interface, context sensitive, GUI
• Task parallelization and workflow management
• Work towards interoperability
• Investigate access methods Obit/AIPS vs casa MS
• Compare calibration model

30
Workpackages

• ParselTongue
• Common invocation methods AIPS/casa/LOFAR/(Miriad)
  
• Interactive user interfaces
• Distributed computing
• High level/task parallelization
• Transforming ALBUS work to casa/LOFAR
• Mosaicing, maybe focal plane arrays
• Global fringe fitting
• Algorithm research
• Common implementation, more than one platform
• Interoperability
• Exercises in data access methods
• AIPS methods on MS structures
• Calibration model transformations

31
How does this fit?

• Participants contacted and positive
• ALBUS group ASTRON/WSRT, JBO, MPIfR, JIVE
• ALMA group NRAO, IRAM, ESO
• LOFAR
• Lots of overlap with some of the other proposals
• ATNF on parallelization and focal plane arrays
• Oxford and Cambridge on parallelization
• Can we also agree on the form of the software
• But other proposals are also interesting and
  feasible
• This looks like 2 x ALBUS in size
• 40 man-years of which 20 contributed
• Could have 6-8 partners
• Including some in non-EC countries
• Amounts to 1.6 2.0 M
• Several scaled down options possible
• But then also scale down number of partners