Archive Activities and Plans STECF Annual Review February 28th 2006

1
Archive Activities and PlansST-ECF Annual
ReviewFebruary 28th 2006

• Richard Hook
• Alberto Micol
• Jonas Haase
• Diego Sforna
• The ST-ECF HLA Team

2
Overview

• Introduction and Background
• Status and Usage
• Developments in 2005
• The Hubble Legacy Archive (HLA) at ST-ECF
• Conclusions

3
Introduction Background

• Maintaining a copy of the Hubble archive is a
  fundamental ST-ECF Role
• The archive has always been a joint ESO/ST-ECF
  activity and most hardware and software is
  currently shared
• Originally the Hubble archive dominated but now
  ESO data is the largest and fastest growing
  component
• Hubble has some unique requirements - such as
  on-the-fly reprocessing (OTFR)
• The ST-ECF, along with CADC and STScI, pioneered
  many new archive features - Preview, OTFR, WFPC2
  associations
• About 35 of Hubble-based publications are NOT
  from the original PI/CoIs.

4
Expected evolution

• The ESO archive is planning for greatly increased
  data volumes (VLTI/VISTA/VST/ALMA/ELT).
• Hubble data volumes will increase by modest and
  predictable amounts, depending on whether SM4
  takes place. Volume is not a problem.
• Both archives are aiming towards the provision of
  high level data products quickly accessible from
  a fast cache and through VO interfaces.
• The requirements of the two archives are
  diverging technically, but many synergies remain.

5
Archive Usage - numbers of requests
Unbiased with internal (eg, ESO QC, ECF
internal projects) requests removed.
6
Archive usage - data volumes
7
Developments in 2005

• Retirement of ageing and unreliable DVD jukebox
• Introduction of temporary magnetic disk based
  storage
• Start of migration to NGAS (ESO standard)
• Continuing work on WFPC2 associations (CADC
  STScI) and initial work on ACS associations
• Involvement with VO work and IVOA standards
• Release of GOODS FORS2 CDF-S spectroscopic data -
  ideal major pathfinder project for HLA.
  Collaboration with ESO and external groups.

8
GOODS Spectroscopy Web page Example -
one Spectrum from more than 1000.
9
The Hubble Legacy Archive (HLA) at ST-ECF

• As Hubble approaches and passes end of life the
  importance of the archive as a research tool will
  increase even further
• Simultaneously expertise in Hubble instrument
  science will decline
• In the more distant future we must ensure the
  long-term accessibility and usability of the
  Hubble legacy

10
HLA CharacteristicsWhat is different from now?

• High-level, science-ready data products
• Best possible calibration
• Full characterisation metadata
• Full documentation
• Physical units (sky position, flux, wavelength)
• Meaningful error estimates
• Typical products
• Combined, cleaned, coadded images
• Extracted spectra
• Access through general purpose interfaces and VO
  protocols
• Almost instant access in most cases

11
HLA at the ST-ECF

• The HLA provides an ideal focus for several areas
  of ST-ECF expertise
• Instrument science and physical modeling
• High-level science product creation (GOODS, FOS
  etc)
• Virtual Observatory expertise and involvement
• Development and management of successful archives
• Continues and deepens successful collaborations
  with other sites interested in HLA (CADC STScI)
• Continues current projects (WFPC2 associations,
  ACS associations, NICMOS ACS grism extraction
  etc)
• Work will concentrate on high-level data product
  creation, characterisation and publishing to the
  VO

12
Current HLA work and plans for 2006

• A pilot project to extract a uniform set of
  NICMOS grism spectra has started
• Exploits unique ECF NICMOS grism expertise
• Will adapt existing, robust, aXe software
• Will validate the full end-to-end HLA concept for
  a small, manageable and scientifically
  interesting data set
• An international HLA collaboration meeting in
  Garching has been organised for 4/5 May 2006

13
NICMOS Grism HLA Pilot Project

• Started in early 2006
• Aim to extract spectra from a uniform set of
  NICMOS G141 grism exposures (about 4000 datasets)
• Initial phase, assessing whether aXe, a robust
  grism extraction tool developed by the ST-ECF for
  ACS, can be adapted for NICMOS, is in progress
• Several NICMOS-specific effects need to be
  handled (tracing spectra, intrapixel sensitivity
  variations etc)
• First results promising

14
Test NICMOS grism exposure
aXe drizzled combined spectrum
G141 NICMOS Grism image of High-z QSO, one of
Four dithered exposures
15
Conclusions

• The archive continues to function very well
• The hardware and software are being adapted in
  view of the evolution of ESO and Hubble
  requirements
• The Hubble Legacy Archive activities are starting
  and form a natural match between the needs of the
  user community and the expertise of the ST-ECF