The AstroWise Pipeline Conceptual Design Review Roeland Rengelink rengelinkstrw.leidenuniv.nl

1
The Astro-Wise PipelineConceptual Design
ReviewRoeland Rengelink rengelink_at_strw.leidenuniv
.nl

• Requirements
• Development Framework
• Prototype

2
Baseline Requirements

• Data Flow System User Requirements (URD)
• overview of the scientific requirements,
• includes calibration and processing requirements
• Date Flow System Calibration Plan (CP)
• detailed description of all calibration
  procedures
• Date Reduction Specifications (DRS)
• specifies the data reduction tasks for the
  calibration and science data

3
Additional Requirements

• Multi-instrument support
• Multi-site operations
• Database Driven operations
• Extendable
• On the fly reprocessing

4
Database Requirements

• Persistence
• Object Oriented DB technology
• Hide schema (code is DB schema)
• Hide ingest/update
• Queries
• Relational DB technology
• Hide SQL
• Dynamic (new)
• User defined objects / tables
• Multi-Site
• Integrated bulk data storage

5
User Interface Requirements

• Developers
• Use pipeline API to write scripts
• Command Line users
• Recipes
• GUI
• Pipeline operations
• Quality Control
• Processing status

6
Pipeline components

• Data reduction recipes
• Data reduction tools
• Infrastructure
• Administration
• User Interface
• Multiple developers
• Multiple users
• Multiple sites

7
Development Framework

• Deliver usable code
• Working
• Understandable
• Correct

8
Python

• Object Oriented Programming
• Modularity
• Reusability
• Rapid Application Development
• Incremental development
• Inline documentation
• Understandable code
• Unit testing framework
• Working code

9
Unit testing

• Write a test for each unit of functionality
• To know that it works
• To reduce the cost of modifying code
• We find bugs (also in eclipse and LDAC)
• But,
• Requires discipline
• 25 overhead

10
CVS

• Essential for collaborative projects
• Initial investment
• A different mind set
• Requires discipline
• Commit/Update early and often
• Only commit working code

11
Astro-Wise Pipeline - Scope

• In
• Calibration operations defined in CP
• Co-added, calibrated, pointings
• Out
• Mosaics
• Masking, other than CCD defects
• Homogenization PSF
• These could be covered in WP2

12
Astro-Wise Pipeline Prototype - Scope

• Reduce WFI data
• Baseline implementation calibration procedures
• Low-level data reduction routines
• Technology demo OO database interface
• Oracle9i back end
• Technology demo parallel processing
• New approach under development
• Establish development framework

13
Astro-Wise Pipeline Prototype - Status

• See CVS
• No photometric calibration yet
• Limited set of recipes
• Database back ends
• Parallel processing
• Documentation