Automated Loon Validation 1

1
Automated Loon Validation

• Nick West

2
Automated Loon Validation What is it?

• It is not verification
• It does not attempt to show that the code is
  giving the right answers.
• Instead it is confirmation that nothing unplanned
  has happened
• i.e. Regression Testing making sure the code
  gives the same answers.
• It has to be extensible. As code develops
• Produces more answers ? need more tests.
• Produces different answers ? need to change
  tests.
• It has to run routinely
• Via a nightly cron job.
• By individuals while testing their code.

3
Automated Loon Validation Why do it?

• For the same reason we do nightly builds
• It saves a great deal of time!
• It saves time identifying problems
• If results have changed since last nightly cron
  then it must have been something related to
  changes in the last 24 hours. Most people can
  remember that far back.
• It saves time finding solutions
• There is less time for concrete to set making
  code changes easier.

4
Automated Loon Validation What do we have right
now?

• Well, not a lot!
• An AFS directory-
• /afs/fnal.gov/files/expwww/numi/html/minwork/co
  mputing/loon_validation_jobs
• Containing
• A few loon scripts and associated data files.
• A perl script that checks a loon log file to see
  that it looks plausible.
• A perl driver script that runs every loon script
  it can find and checks the log file each
  produces.
• It is extensible
• Add new loon scripts and driver will run them
  automatically
• But checking a loon job doesnt crash is hardly a
  validation!
• It needs to be developed!

5
Automated Loon Validation What do we do next?

• We need to check the answers
• Produce ntuples and compare statistically to
  reference versions.
• Where ntuples are not appropriate e.g. framework
  services such as DBI provide stand alone tests
  and integrate in.
• When the answers change
• If unexpected find and fix problems.
• If expected verify and update reference.
• Problem isolation
• Changes upstream can produce validation
  failures downstream.
• Need to slice up the job path with standard
  inputs at each stage.
• Support distribution
• No one person can do everything!
• Ideally-
• Every package owner is responsible for providing
  a validation script.
• A nightly cron publishes results on the web.
• One person polices and chases owners of failing
  package.

6
Automated Loon Validation What Framework?

• What we have right now is inadequate
• Only works for loon jobs and then only to check
  they dont crash!
• It doesnt work for standalone validation.
• We also require targeted validation, allowing a
  package owner just to validate their package.
• Should we use/develop something based on Paul
  Echevarrias Validation package?
• Worked on 3 years ago.
• Never used in anger.

7
Automated Loon Validation Finally

• Sorry, I cannot resist
• I have been bleating on about this for years
  (e.g. S/W call 10 July 2003)
• I have my own systems
• Standalone testing of DBI, NAV etc.
• In SNO a very basic system that checks 130
  ntuples and takes about 3 minutes to run.
• Automated validation is extremely useful