ESO Reflex A Graphical Workflow Engine for Data Reduction Richard Hook Euro VO Data Centres Alliance

1
ESO ReflexA Graphical Workflow Engine for Data
ReductionRichard HookEuro VO Data Centres
Alliance Theory Grid Workshop,Garching, April
2008
2
Overview

• The Sampo Project
• The ESO data reduction context
• ESO Reflex
• Examples
• Future plans
• Apology this talk is not really about grid
  activities (and definitely not about theory)

3
Sampo overview

• As part of Finlands joining fee for ESO a
  contribution in kind of computer scientist
  staff was made available and named Sampo.
• Sampo started in January 2005 and ended in
  January 2008
• The aim of the project was to assess the
  requirements for ESO data reduction and analysis
  software infrastructure in the medium term and
  perform a series of pilot projects to assess
  different options and produce useful tools
• The project was managed at ESO Garching with the
  team based in Helsinki, Finland

4
The Sampo priorities

• Enabling and facilitating science-grade reduction
  of ESO data from the La Silla Paranal Observatory
  was identified as its primary goal of the Sampo
  project
• Long-standing request from the community
• Input from Instrument Scientists in Garching and
  Chile
• Sampo SAC
• Sampo has concentrated on developing ESO Reflex,
  a graphical user interface to run ESO data
  reduction recipes
• Other sub-projects conducted by Sampo
• PyMidas a Python interface to Midas
• VODA a pilot project addressing the integration
  of data analysis environments and the Virtual
  Observatory (cancelled)

5
The data reduction challenge

• ESO cannot reduce all of the data its telescopes
  and instruments produce to a level where their
  full scientific potential will be exploited
• The responsibility for the quality of the
  scientific reduction of the data can only rest
  with the individual users
• The users are, then, faced with the challenge of
  a timely and accurate data reduction
• Proprietary and/or archival data from several
  instruments often need to be combined
• As volume and complexity of data increase, a
  detailed knowledge of the instrument, data format
  and header content is essential to fully exploit
  the data
• Science-quality pipelines are increasingly needed
  also for Quality Control
• General-purpose tools like IRAF and ESO-MIDAS are
  inadequate for the task and instrument specific
  software, implementing carefully tuned algorithms
  is essential

6
Data reduction by individual users the context

• ESO provides pipeline recipes for all VLT/VLTI
  instruments
• They remove the instrumental signature and are
  used for quality control at ESO and distributed
  to the community
• In some cases the data products are adequate for
  scientific analysis, but this is generally not
  yet the case
• Offline tools (Gasgano and EsoRex) are available
  to call the pipelines, but lack some of the
  functionalities needed by the community
• AND
• Many older general purpose reduction and analysis
  systems remain in wide use (MIDAS/IRAF etc.) as
  they contain valuable algorithms
• Many instrument-specific packages have been
  developed in the community (e.g., Euro3D tools,
  VIPGI etc)
• Greater use will be made of remote data resources
  and the Virtual Observatory (VO)

7
Introducing ESO Reflex a graphical data
reduction environment

• A data reduction system for the end user
  requires
• Modular recipes to provide access to intermediate
  products
• Interactive tools, defined or customized by the
  user, to analyze intermediate and final data
  products
• A user-friendly, intuitive and flexible interface
• The ESO Reflex tool, addresses the interface
  issue, with a focus on the use case of ESO data
• Dedicated invoker for CPL-based recipes
• General invoker for Python scripts (hence PyRAF
  PyMidas)
• General invoker for IDL scripts
• Invoker for the command line
• Many other Taverna features for free

8
ESO Reflex look and feel
ESO Reflex is based on Taverna, a popular open
source Java workflow engine
9
Main features of ESO Reflex (over and above what
Taverna offers)

• In interactive mode the user can make changes to
  input data and parameters during execution
• Errors on during recipe execution are detected by
  ESO Reflex and appropriate action can be taken
• (Some) flow control looping, skipping,
  conditional statements, etc.
• Parallel execution full advantage on
  multi-processor or multi-core machines
• Customisability workflows are easily modified,
  Python and IDL interfaces, system commands can be
  invoked, easy access to VO and other web
  services
• FITS file handling (using Gasgano code) data
  organization, tagging, selection

10
ESO Reflex in action FORS2 MXU
11
ESO Reflex in action a FORS Calibration
workflow.
12
ESO Reflex and VO services

• Reflex/Taverna works very well with distributed
  resources and web services
• Reflex also supports the PLASTIC protocol for
  passing information between client-side VO tools
• As a small test project a workflow has been
  developed that finds VO image data, passes it to
  a remote processing server and uses PLASTIC tools
  running locally
• Reduction of local ESO data and access to remote
  VO facilities within same environment

13
(No Transcript)
14
Potential shortcomings of ESO Reflex

• Yet another tool to learn, support etc.
• Not as powerful as a script
• Flow control limited (loops not easy)
• Not suitable for very complex workflows

15
The bottom line on ESO Reflex

• The Sampo project has enabled ESO to explore
  options for future data reduction systems
• The ESO Reflex model is cost effective because it
  capitalises on ESOs investment in CPL-based
  algorithms
• Offering ESO pipelines to the community in a more
  flexible way
• Stimulating more scientific feedback from the
  community and, therefore, steering algorithm
  development towards more science-grade
  applications
• It exploits existing systems to provide those
  facilities that CPL is not designed to offer
  (e.g., graphics, interactivity, etc.)
• It avoids the need to develop from scratch a
  multi-purpose, and expensive, full-fledged data
  reduction environment

16
The future of ESO Reflex and pipelines

• The Sampo project itself ended in January 2008
• ESO has examined the outcome of Sampo and
  concluded that
• ESO Reflex is a viable way of addressing some of
  the mid-term needs of the user community
• The continued development of science-grade
  CPL-based data reduction recipes is vital
• The development of interactive tools is also
  needed

17
Status and plans

• ESO Reflex V1.1 is currently available as a
  beta-test version on request
• Current version is built on Taverna 1.5 -
  conversion to Taverna 2.0 will be needed, and
  involves significant changes
• API changes
• Use of Maven
• Some UI refactoring
• A public release is planned for
  late-2008/early-2009