A bottom-up approach to data calibration Carlo Izzo

1
A bottom-up approach to data calibrationCarlo
Izzo
2
MOSES

• Multi
• Object
• Spectroscopy
• Empirical
• Self-calibration

Many MOS
MOSes!
3
Major requirements for automatic data reduction
Robustness

• managing unexpected situations
• supplying information about what went wrong
• actual monitoring of the instrument

Flexibility

• using general algorithms
• applying instrument-independent DRS strategies
• low maintenance cost

4
Major requirements for automatic data reduction
Google search flexible software reusable
software robust software working
software bug-free software

• 551,000 hits
• 495,000
• 353,000
• 223,000
• 56,000

Robustness

• managing unexpected situations
• supplying information about what went wrong
• actual monitoring of the instrument

Flexibility

• using general algorithms
• applying instrument-independent DRS strategies
• low maintenance cost

5
Identification of reference objects
Reference objects are used for data calibration

• Stars
• astrometric calibration
• photometric calibration
• Spectral lines (arc lamp, sky)
• wavelength calibration
• Spectral edges (flat)
• spectral curvature
• Slit positions (pinhole mask)
• mask-to-CCD transformation

6
Identification of reference objects
How to identify reference objects? The top-down
approach
The model
The catalog
The objects
The improved model
7
(No Transcript)
8
Identification of reference objects
How to identify reference objects? The top-down
approach
The model
The catalog
The objects
The improved model
9
A typical MOS arc lamp exposure
10
Using first-guess model to find reference lines
11
Earthquake!
12
False matches confirm expectations
VIMOS-MOS, LR_red, all quadrants
13
Finding standard stars
14
Finding standard stars
ERROR no standard star found cannot compute
zeropoint
15
Advantages of the top-down approach

• The safest approach for stable instruments
• The only possible approach if very few reference
  objects are available

16
Disadvantages of the top-down approach

• Not robust it fails with unstable instruments
• Not flexible it requires frequent
  reconfiguration
• Biased toward expectations
• Misled by contaminations

17
Finding standard stars
The stars
The catalog
18
Finding standard stars
The catalog (i.e., the pattern)
19
Finding standard stars
The stars
The catalog (i.e., the pattern)
20
Finding standard stars
The stars
The catalog (i.e., the pattern)
21
Finding standard stars
The stars
The catalog (i.e., the pattern)
22
Point pattern-matching (2D)
G. S. Cox et al. (1991) A New Method of
Rotation, Scale and Translation Invariant Point
Pattern Matching Applied To the Target
Acquisition and Guiding of an Automatic
Telescope
23
Looking for peaks

• ______________________________________

24
Looking for patterns

• The pattern wavelengths
• 5400.562
• 5460.742
• 5764.419
• 5769.598
• 5790.656
• 5852.488
• 5875.620
• 5881.900

• The data pixel positions
• 1220.64
• 1253.23
• 1299.44
• 1304.07
• 1339.30
• 1400.33
• 1450.28
• 1457.32
• 1471.00
• 1496.21

25
Looking for patterns

• The pattern wavelengths
• 5400.562
• 5460.742
• 5764.419
• 5769.598
• 5790.656
• 5852.488
• 5875.620
• 5881.900

• The data pixel positions
• 1220.64
• 1253.23
• 1299.44
• 1304.07
• 1339.30
• 1400.33
• 1450.28
• 1457.32
• 1471.00
• 1496.21

26
A simple case calibrating a single spectrum

• _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

27
Identifying arc lamp lines
28
Wavelength map

• Mean accuracy 0.05 pixel

29
Resampled spectrum
Mean accuracy 0.05 pixel
30
A case with many spectra (VIMOS GRIS_HRred)
31
Wavelength map
Mean accuracy 0.07 pixel
32
Another example (FORS2-MXU GRIS_150I)
33
Wavelength map

• Mean accuracy 0.05 pixel

34
Identifying the spectra

• Select the reference wavelength in this
  example, l 7000.00 A

35
Identifying the slits
The mask

• Select the reference wavelength in this
  example, l 7000.00 A

The CCD
36
Measuring the spectral curvature
37
Measuring the spectral curvature
38
Rectified image
39
Rectified image
40
Identification of reference objects
How to identify reference objects? The bottom-up
approach
The generated model
The pattern
The candidate objects
41
Identification of reference objects
How to identify reference objects? The bottom-up
approach
The generated model
The pattern
The candidate objects
42
Robustness

• This approach can cope with unexpected position
  and/or number of spectra
• Reference lines are more safely identified for
  being part of a pattern, rather than for being
  close to some expected position

43
Flexibility

• This approach is (MOS) instrument-independent
• Low maintenance cost, even at instrument upgrades
  (new chip, new grism, new lamps)
• This approach may be applied to extracted
  products from any kind of spectroscopic data (not
  just MOS, but also IFU, echelle, etc.).

44
GIRAFFE Medusa1_H525.8nm
Mean accuracy 0.10 pixel
45
Disadvantages of the bottom-up approach

• As the top-down approach depends on a model, the
  bottom-up approach depends on the data!
• This approach is a black box as for any program
  based on a bottom-up strategy (e.g., using
  trained neural networks), it is often difficult
  to find the reason of a failure.

46
The CRIRES data reduction challenges

• Inputs
• The Estimate Polynomial
• The Wavelength Error of the estimate (WLerror)
• The degree of the searched polynomial (degree)
• The number of samples (nsamples)
• The lines catalogue (OH, Gas cell, Lamps, Hitran)
• The signal to calibrate (in pixels)

• Algorithm
• Consider degree1 positions Ai regularly spaced,
  and nsamples points spread within WLerror around
  these positions
• For each possible sequence of points
  (nsamples(degree1) possibilities), the
  interpolation polynomial is created and
  considered as candidate
• The candidate polynomial is used to convert the
  signal to calibrate from pixels to wavelengths.
  This signal is compared to the signal generated
  from the catalogue. A likelihood coefficient in
  computed
• The best likelihood parameter gives the best
  candidate, i.e. the polynomial that is the
  closest to the solution
• A second pass (or more) is used to refine the
  solution with the first pass solution used as
  estimate, with a smaller WLerror and a higher
  degree.

47

• Inputs
• The Estimate Polynomial
• The Wavelength Error of the estimate (WLerror)
• The degree of the searched polynomial (degree)
• The number of samples (nsamples)
• The lines catalogue (OH, Gas cell, Lamps, Hitran)
• The signal to calibrate (in pixels)

• Algorithm
• Consider degree1 positions Ai regularly spaced,
  and nsamples points spread within WLerror around
  these positions
• For each possible sequence of points
  (nsamples(degree1) possibilities), the
  interpolation polynomial is created and
  considered as candidate
• The candidate polynomial is used to convert the
  signal to calibrate from pixels to wavelengths.
  This signal is compared to the signal generated
  from the catalogue. A likelihood coefficient in
  computed.
• The best likelihood parameter gives the best
  candidate, i.e. the polynomial that is the
  closest to the solution
• A second pass (or more) is used to refine the
  solution with the first pass solution used as
  estimate, with a smaller WLerror and a higher
  degree

48
CRIRES Wavelength Calibration

• Inputs
• The Estimate Polynomial
• The Wavelength Error of the estimate (WLerror)
• The degree of the searched polynomial (degree)
• The number of samples (nsamples)
• The lines catalogue (OH, Gas cell, Lamps, Hitran)
• The signal to calibrate (in pixels)

• Algorithm
• Consider degree1 positions Ai regularly spaced,
  and nsamples points spread within WLerror around
  these positions
• For each possible sequence of points
  (nsamples(degree1) possibilities), the
  interpolation polynomial is created and
  considered as candidate
• The candidate polynomial is used to convert the
  signal to calibrate from pixels to wavelengths.
  This signal is compared to the signal generated
  from the catalogue. A likelihood coefficient in
  computed
• The best likelihood parameter gives the best
  candidate, i.e. the polynomial that is the
  closest to the solution
• A second pass (or more) is used to refine the
  solution with the first pass solution used as
  estimate, with a smaller WLerror and a higher
  degree.

49
Thanks to
Michele Peron (ESO/SDD) Sabine Moehler
(ESO/DFO) Pascal Ballester (ESO/SDD) Lars Lundin
(ESO/SDD) Kieran OBrien (ESO/PSO) Emmanuel Jehin
(ESO/PSO) Ralf Palsa (ESO/SDD) Marguerite Pierre
(CEA, Saclay, France) Stefano Cristiani (INAF,
Trieste, Italy) Christophe Adami (LAM, Marseille,
France) Harald Kuntschner (ESO/ST-ECF) Martino
Romaniello (ESO/DFO) Vanessa Doublier
(ESO) Sandro Villanova (DdA, Padova,
Italy) Stefano Bagnulo (ESO/PSO) Burkhard Wolff
(ESO/DFO) Emanuela Pompei (ESO/LSO) Ivo Saviane
(ESO/LSO)
50
Thanks to

• - prime mover
• supervisor
• quality control (FORS)
• image processing advisor
• math advisor
• requirements
• requirements
• software advisor
• astronomer advisor
• astronomer advisor
• beta-tester (FORS)
• astronomer advisor
• astronomer advisor
• beta-tester (FORS)
• beta-tester (FORS)
• astronomer advisor
• quality control (VIMOS)
• requirements (EMMI)
• requirements (EFOSC)

Michele Peron (ESO/SDD) Pascal Ballester
(ESO/SDD) Sabine Moehler (ESO/DFO) Yves Jung
(ESO/SDD) Lars Lundin (ESO/SDD) Kieran OBrien
(ESO/PSO) Emmanuel Jehin (ESO/PSO) Ralf Palsa
(ESO/SDD) Marguerite Pierre (CEA, Saclay,
France) Stefano Cristiani (INAF, Trieste,
Italy) Christophe Adami (LAM, Marseille,
France) Harald Kuntschner (ESO/ST-ECF) Martino
Romaniello (ESO/DFO) Vanessa Doublier
(ESO) Sandro Villanova (DdA, Padova,
Italy) Stefano Bagnulo (ESO/PSO) Burkhard Wolff
(ESO/DFO) Emanuela Pompei (ESO/LSO) Ivo Saviane
(ESO/LSO)
51
Looking for patterns

• The pattern wavelengths
• 5400.562
• 5460.742
• 5764.419
• 5769.598
• 5790.656
• 5852.488
• 5875.620
• 5881.900

• The data pixel positions
• 1220.64
• 1253.23
• 1299.44
• 1304.07
• 1339.30
• 1400.33
• 1450.28
• 1457.32
• 1471.00
• 1496.21

52
Looking for peaks