Trilinos SQA Training

1
Trilinos SQA Training

• Robert Heaphy
• October 16, 2003

2
ASCI Program Quality Policy

• ASCI Program Plan,  DOE/DP-99-000010592
• DOE Notice 203.1, Software Quality Assurance
• DOE Order 414.1A
• Quality Criteria, QC-1,  NNSA/AL
• ASCI Software Quality Engineering Goals,
  Principles, and Guidelines DOE/DP/ACI-SQE-2000-FDR
  FT-VERS2  (commonly called the Trilab GPG) 
• Sandia National Laboratories ASCI Applications
  Software Quality Engineering Practices (A version
  has appeared for each assessment year)

3
Sandia Corporate Quality Policy

• DOE Order 414.1A
• 10 CRF 820
• Quality Criteria, QC-1,  NNSA/AL
• (Sandia) Corporate Quality Assurance Program 
  CPR001.3.2
• (Sandia) Corporate Software Quality Assurance 
  CPR001.3.6
• QC-2  (which specifically allows the use of QC-1
  instead)

4
Sandia Quality Home Page

• Pubic Law 104-113    (Federal agencies should now
  use non government standards)
• DOE order accepting the use of non DOE standards
• ISO 9000   (Standard of choice)            

5
Other Quality Standards at Sandia

• CMM   (DOD) replacing MilSpec 2167A
• TSP, PSP, P-CMM   
•                
• CMMI (to replace obsolete CMM)
• Baldrige

6
Last Audit Results

• Final report being written 3 weeks ago and is not
  yet available.
• Trilinos and Zoltan did well!
• Primary problem was missing processes (such as
  training)
• Next audit may have changes, but no indications
  of future directions yet
• Management is concerned about next ASCI external
  audit (FY 2005?)

7
Trilinos SQE Program

• Assumption There is a true and significant
  component of research in Trilinos.
• Assumption Professionally, you already know how
  to perform your job well and are doing it.
• You dont need 400 practices in CMM or 618
  pages of CMMI!
• You need to get credit for what you already do.
• This is a generic quality program based on QC-1.
• GOAL To continue a successful quality program
  which is minimally intrusive to developers and
  provides tangible benefits the development team.
• KEY Document our existing work practices using
  lightweight process checklists, then follow the
  checklists.

8
Changes for this next audit cycle

• Improve, complete and use process checklists.
• Implement checklist tracking (team leaders).
• Suggest the use of laboratory notebooks project
  notebooks.
•   
• Write new version of Trilinos ( Zoltan) SQE/SQA
  documentation (Heaphy).
• Each package team will perform a Trilinos self
  assessment.
• Future changes (TBD)
•  traceability from requirements to maintenance 
  (QFD, perhaps).
•  continuous process improvement (PDCA/PDSA).

9
Suggestion

• Avoid negative vocabulary in emails, documents,
  CVS comments, etc.
• Specifically, think how ASCI management (who may
  not understand programmer jargon) might react to
  terms like hack, trick, and work around.

10
Process Checklists

• Include necessary boilerplate from Trilinos SQA
  examples
• Divide each checklist into 5-10 key activities. 
  Provide a simple explanation of the activity and
  what is to be accomplished.
• Incorporate creating and saving appropriate
  artifacts
• Incorporate Peer Reviews and double checks
• Incorporate standard tools (Bugzilla, Doxygen,
  Bonzai, Purify, etc)
• Incorporate a few, simple metrics as appropriate
• Incorporate error proofing (future)
• Maintain master checklist under CVS control
• Maintain specific instantiation under CVS control
• Don't commit yourself unnecessarily! (Conduct
  code review vs conduct Fagan Review. Is a team
  code review invalid if a single team member is
  missing? Do you really want require a "Coding
  Style Guide"?)

11
Process Checklists - 2

• Follow the checklist every time.  If not
• Change the checklist through a formal process. 
  Do this to add improvements,  remove ambiguity,
  modify processes, etc.
• Obtain a formal waiver (please don't really do
  this!)
• Document procedure changes in lab notebook for
  RD activities per QC-1! (A changed procedure
  must be signed and dated)
• Otherwise, failing to follow the checklist is a
  significant problem and a potential audit
  finding.
•         
• NEW Track using spreadsheet/database (saved in
  CSV (ASCII) text for CVS)

12
New Training Checklist -1

• New Developers (within 1 month of start)
• Read "An Overview of Trilinos"
• Read "Trilinos Users Guide"
• Read "Trilinos Developers Guide"
• Read "Trilinos Developers Guide, Part 2, ASCI
  SQA"
• Read Package Developers kit
• Mentor Assignment with initial 2 hours mentoring
  (provide dates/times with mentor name)

13
New Training Checklist -2

• All Developers (three year continuous training
  cycle)
• Over each three year cycle, each developer must
  receive and document the indicated training
  hours.  One hour credit is given for an hour of
  formal class attendance, conference attendance,
  team/project training, mentoring (either as
  mentor or student).  One hour credit is given for
  participation in the monthly Trilinos phone
  conferences. Three hours credit will be given for
  a published paper (author or coauthor),
  conference presentation, or for each hour of
  instructing a relevant training course.  Each
  developer will list below at least the required
  hours of training and maintain supporting
  documentation.     
• Math and Algorithms training (60 hours)
• SQE, SQA training  (15 hours)
• Development tools training (5 hours)