Retooling Data Centre Infrastructure to Support the Virtual Observatory

1
Retooling Data Centre Infrastructure to Support
the Virtual Observatory

• Séverin Gaudet

2
Current Collection
Dominion Radio Astrophysical Observatory (DRAO)
Penticton
Hubble Space Telescope (HST)
Far Ultraviolet Space Explorer (FUSE)
Canada-France-Hawaii Telescope (CFHT) Hawaii
Gemini North Telescope Hawaii
Microvariability and Oscillations of Stars
Telescope (MOST)
James Clerk Maxwell Telescope (JCMT) Hawaii
Gemini South Telescope Chile
3
Growth

• Received gt 61 Tbytes in 2005
• 7 sources of data
• Network transfer only

4
Delivery

• Delivered gt 38 Tbytes in 2005
• Provided data and services to gt 2500 distinct
  addresses
• Network distribution only

5
Current Collection

• Heterogeneous collection
• 10 year old archive model
• Evolved
• Evolving

6
Evolution
7
Evolution
8
New Projects ? New Requirements

• New Projects, New Funding
• JCMT new instrumentation, ADP, access, VO
  presence
• Gemini ADP, access, VO presence
• CFHT ADP, access, VO presence
• HST Legacy Archive, ADP, VO presence

9
New Projects ? New Requirements

• New Requirements
• Proprietary data access
• Survey team and PI support
• Collaboration support
• Advanced products
• Versioning
• Data relationships
• Duplicate photons
• Data packaging
• Data cache
• Programmatic access to data
• Externally produced ADP
• VO presence

10
VO view in 2003
Data Warehouse

• Archives
• Archive metadata
• Process control
• Storage control

• Processed products
• Advanced data products (ADP)

11
Data Transformations
12
Data Transformations
VO Interfaces
Common Archive Observation Model
VO Views (e.g. SIA, Octet)
Common Archive Observation Model
Telescope Instrument Data Model
Telescope Instrument Data Model
Telescope Instrument Data Model
Telescope Instrument Data Model
Archive Specific Meta Data
Archive Specific Meta Data
Archive Specific Meta Data
Archive Specific Meta Data
Archive Interfaces
Archive Interfaces
Archive Interfaces
Archive Interfaces
13
Common Archive Observation Model

• To be implemented in each archive
• Merged in the data warehouse
• Purpose
• Standardize the core of every archive
• The only metadata interface between archives and
  the data warehouse
• A general purpose infrastructure to respond to
  evolving VO standards
• Model
• Inspired from VO work Observation,
  Characterization, SIA, SSA, Authentication, etc.
• Using our archive, data modelling and VO
  experience
• Characterisation based on FITS WCS papers I, II
  and III

14
Other Models Affected

• Storage Model
• Increased storage capacity
• Affects the low level storage access e.g. adGet
  -a CGPS -s -op getData?cutout100100,210210,12
  72,11 CGPS_MA2_HI_line -nocrc -o -
• Processing Model
• Process and store everything
• Systematic generation and re-generation of
  products
• Retrieval Model
• Complex data packages
• Different access methods
• Logging Model
• Access Control Model
• PI, Survey teams, Collaborations

15
The Path Forward

• Retooling the CADC from top to bottom
• Horizontal development
• Common infrastructure
• Design process began in the fall of 2005
• The challenge and the risk
• Fundamental changes
• Everything happening at once!
• Initial buy-in to VO may be small but a full
  commitment is not!

16
The Path Forward

• For archives
• JCMT
• Under design
• Requires an implementation Fall 2006
• Gemini, HST CFHT
• Design to begin Fall 2006
• Implementation Spring 2007
• For CVO
• Harvesting tools to be developed
• Views to support IVOA models (e.g. SIA 1.1, SSA
  1.0, Cone search,)

17
(No Transcript)