Test

1
Test Analysis Projectaka statistical testing

• Maria Grazia Pia, INFN Genova
• on behalf of the TA team

http//www.ge.infn.it/geant4/analysis/TandA
Geant4 Workshop, CERN, 2 October 2002
2
What is the Test Analysis project?

• Test Analysis is a project to develop a
  statistical analysis system for usage in Geant4
  testing

• Provide tools to compare Geant4 simulation
  results with reference data
• equivalent reference distributions (for
  instance, regression testing)
• experimental measurements
• data libraries from reference distribution
  sources
• functions deriving from theoretical calculations
  or from fits

3
History

• Statistical testing agreed as a
  collaboration-wide goal 2001-2002
• Initial ideas for this project presented at a TSB
  meeting end 2001
• LowE Anaphe expressed interest to work at the
  project, Nov 2001
• Informal discussions with STT Coord. Gabriele
• Informal discussions with e.m. physics coord.
  developers
• Open brainstorming workshop 31 May 2002
• Vision User Requirements Document
• http//www.ge.infn.it/geant4/analysis/TandA
• Open discussion and agreement on vision and
  requirements
• at this workshop

4
Vision the basics

• Have a vision for the project
• A general tool for physics STT?
• Only statistical testing? in STT?
• Other parties than Geant4 interested?

Clearly define scope, objectives

• Who are the stakeholders?
• Who are the users?
• Who are the developers?

Clearly define roles

• Rigorous software process
• no cow-boy process please

Software quality

• Build on a solid architecture
• OOAD, not just hacking some code

Flexible, extensible, maintainable system
5
The problem we want to solve (1)
6
The problem we want to solve (2)
7
The scope of the project

• The Test Analysis project will provide tools
  for statistical testing of Geant4 physics and
  related quantities
• both at unit and system level
• It will address primarily the areas of physics
  validation and regression testing
• by means of comparison between simulation results
  and reference distributions
• Generality (for application also in other areas)
  should be pursued

8
Guidelines

• The TestAnalysis tool adopts a solid
  architectural approach and a rigorous software
  process
• to offer the functionality and the quality needed
  by the users
• to be maintainable over a large time scale
• to be extensible, to accommodate future
  evolutions of the requirements
• Alternatives
• hacking code for ad-hoc solutions
• Pro some specific problems solved quickly
• Contra difficult to cover all requirements, not
  easily maintainable or extensible
• a project limited to statistical regression
  testing for the STT
• Pro some useful results in a critical area
• Contra
• not encompassing the testing requirements
  specific to physics Working Groups
• by neglecting the commonality of some problems,
  it would not optimise the exploitation of the
  resources

9
Who are the stakeholders?
10
Who are the users?

• Other potential users
• users of the Geant4 Toolkit, wishing to verify
  the results of their applications with respect to
  reference data or their own experimental results
• other projects with similar requirements, or
  using the Geant4 Toolkit
• (e.g. the LHC Computing Grid project)

11
What do they want?
See the URD for specific requirements
12
Are there any constraints?

• To operate in the same environment as the
  development of Geant4
• Possibility to integrate it in system testing
• Based on AIDA
• No concrete dependencies on specific AIDA
  implementations should appear in the code of the
  system tests
• User-friendly interface, to make the system easy
  to use and customise even by non-expert users

13
Prototype(s)

• Useful to address some risks
• the viability of the product being developed
• some technological concerns, like the usage of
  Python and the adequacy of AIDA implementation
• the understanding of requirements
• Useful to developers to get familiar with tools,
  environments etc.
• Simple prototype for comparison between NIST
  reference data of electron range and Geant4
  LowE/standard simulation

Risk list at http//www.ge.infn.it/geant4/analysis
/TandA
14
Prototype (S. Guatelli, A. Pfeiffer)
The reference data are automatically downloaded
from the NIST website The user can select the
physical quantity to be tested (range, stopping
power etc.), the type of material and the
energies of the electron The user can visualise
the reference data The user can load the
simulation results from a file and visualise
them The user can overlay simulated and reference
data, with their respective errors, on the same
plot The user obtains the difference () between
simulated and reference data, for all the
energies selected, and plots it together with the
errors of the reference data
Al
15
Related sessions

• Tuesday 14-15.30
• Benchmark statistical tests automation and
  infrastructure issues
• Tuesday 16-17
• Statistical tests review of vision and
  requirements
• Thursday 16.00-18.15
• Benchmark and statistical tests summary, work
  plan
• Goals
• Agree on vision for the project, agree on initial
  set of requirements
• Agree on objectives, milestones, activities,
  responsibilities, schedule