eScience Experiences: Software Engineering Practice and the EU DataGrid

1
e-Science ExperiencesSoftware Engineering
Practice and the EU DataGrid

• Lee Momtahan and Andrew Martin
• Oxford University Software Engineering Centre

2
Contents

• EU-DataGrid
• Challenges
• Comparisons
• Conclusions

3
EU-DataGrid

• 9.8m Euro project over 3 years
• 21 partners in 15 countries
• application in particle physics (and
  bioinformatics, and earth sciences)
• PetaBytes of data datasets to be catalogued,
  replicated where necessary
• seamless delivery of computing resources
• 200 staff, meeting infrequently (60 FTE)

4
Project Goals

• build application frameworks potentially
  involving huge amounts of data, compute power and
  distribution
• provide secure, managed, uniform access to such
  resource
• facilitate collaboration, and remote access to
  data and scientific instruments
• manage such facilities as a persistent service

5
Work Package Structure
6
Our role

• becoming involved after project started
• funded to bring computer science/software
  engineering experience to the project
• intending to help by modelling aspects of design
  in order that the system may be better
  understood, designed, built, documented
• in passing made the observations documented here
• interested in the generality of these issues for
  e-Science

7
Challenges

• Requirements Volatility
• Novel paradigm New diversity Volatile
  off-the-shelf components
• Geographical Separation
• communication can easily become a limiting factor
  (Brookes) Physicists are used to collaborating
  in experiments but software?
• System Decomposition
• Political concerns geographic determination

8
Challenges

• Project Processes and Authority
• there is a quality plan how do you get people to
  follow it? is a commercially-based process
  appropriate? what about traditional academic
  means of QA?

9
Challenges

• Planning and tracking
• exit criteria for an iteration seems to be the
  completion of a document detailing the problems
  found in testing

10
Comparisons

• academic software production vs. commercial
  software production
• academic software production vs. other academic
  activity
• CMM Level for Software? For Paper/Proposal
  Writing?
• open source software vs. open source development
• open source models vs. publicly-funded research
• publication in journals vs. publication to a
  repository

11
Conclusions

• Practice of software engineering in an e-Science
  context is substantially different to industrial
  practice
• Industrial models do not seem appropriate
• Open source models seem to fit better
• Publication and review are the key to quality and
  process improvement