WrapUp

1
Topics
1
Introductions Scenario Introduction Overview
Step 1 Decide to Change Step 2 Get
Sponsorship Step 3 Where Are We? Step 4 Plan
the Change Step 5 Implement the
Change 5a Implement the ChangeTraining 5b
Implement the ChangeQA Step 6 Measure the
Change Training Wrap-Up
Topic 1 Topic 2 Topic 3 Topic 4 Topic
5 Topic 6 Topic 7 Topic 8 Topic 9 Topic
10 Topic 11
2
3
4
5
6
7
8
9
10
11
2
Topic Objectives

• For this section, we assume that a contractor is
  developing the system, and we are interested in
  doing QA on that vendor
• Discuss what is QA?
• Industry standards for QA
• Discuss the basic elements of a sound QA contract
• What goes into it?
• Alternatives to a QA vendor
• Cost of QA

3
What is QA?

• Process and Product Quality Assurance involves
  the following
• Objectively evaluating performed processes, work
  products, and services against the applicable
  process descriptions, standards, and procedures
• Identifying and documenting noncompliance issues
• Providing feedback to project staff and managers
  on the results of QA activities
• Ensuring that noncompliance issues are addressed
• Source Software Engineering Institute.
  Capability Maturity Model Integration (CMMISM),
  Version 1.1 (CMU/SEI-2002-TR-001). Pittsburgh,
  Pennsylvania Software Engineering Institute,
  Carnegie Mellon University, 2000.

4
Process Quality Assurance

• Answers three questions
• Are the vendors processes defined?
• Are the vendors processes adequate to perform on
  the contract?
• Are the vendors activities following these
  defined processes?

5
Management Process

• Are the vendors processes defined?
• Processes include management, support, and
  technical processes
• Management includes
• Project Planning
• Project Monitoring and Control
• Supplier (Subcontractor) Management
• Risk Management

6
Support Process

• Processes include management, support, and
  technical processes
• Support includes
• Configuration Management
• Process and Product Quality Assurance
• Measurement and Analysis

7
Technical Process

• Processes include management, support, and
  technical processes
• Technical (engineering) includes
• Requirements Management
• Requirements Development
• Technical Solution
• Design, implementation, etc.
• Product Integration
• Verification
• Does the product meet its specification
• Validation
• Can the product be used as intended

8
Vendor Process Adequate

• Are the vendors processes adequate to perform on
  the contract?
• QA needs to base their opinion on a standard, not
  just their personal opinion
• Four major standards in the Industry
• CMM/CMMI
• IEEE
• PMI
• ISO 9000

9
CMM/CMMI

• First used in the mid 1980s
• CMM developed for DoD by the Software Engineering
  Institute (SEI) at Carnegie Mellon University
• CMMI included industry leaders as authors

10
CMM/CMMI Pros and Cons

• Pros
• Model is downloadable
• Standard process for evaluating vendors
• Assessors are sanctioned by SEI
• Training available on the model
• Includes all processes
• Cons
• Not recognized by some commercial vendors
• Requires significant training to interpret the
  model
• Assumes that the reader is a software/system
  expert
• Does not include Human Resources (staffing)

11
IEEE

• Formed in the 1960s
• Originally to develop standards for electrical
  components
• Developed a set of standards that include project
  management
• Standards are for purchase

12
IEEE Pros and Cons

• Pros
• Widely recognized
• Standards easily available
• Includes all processes
• Cons
• Uses a specific vocabulary
• Standards are for purchase
• Assumes a standard life cycle
• Does not include Human Resources (staffing)

13
PMI

• Set up to certify Project Managers
• A defined set of processes
• Project Managers Book of Knowledge (PMBOK)

14
PMI Pros and Cons

• Pros
• A good process for certifying Project Managers
• Widely recognized
• Local chapters of PMI
• Includes Human Resource (staffing)
• Certifies Project Managers
• Special interest groups, IT, construction,
  engineering, etc.
• Requires ongoing training to maintain
  certification
• Cons
• PMBOK not set up to evaluate contractors
• Includes only the management and support processes

15
ISO

• Originally used as a method of certifying
  manufacturing plants
• Used extensively in Europe
• Also certifies software developers

16
ISO Pros and Cons

• Pros
• Standard process for evaluating vendors
• Evaluators are sanctioned by ISO
• Certification must be kept up to date
• Cons
• Not recognized by some American vendors
• Requires significant training to interpret the
  model
• Assumes that the reader is a software/system
  expert
• Most documentation is difficult to understand and
  interpret

17
Doing QA From Within

• Internal
• Have your own people trained in doing QA
• Using external standards
• Require vendor certification
• CMMI
• ISO
• PMI

18
Internal QA

• Internal QA
• Set up internal Project Management Office
• Train participants in the appropriate standard
• Have members of the office monitor vendors
• Monitor each deliverable for compliance to the
  standard

19
Internal QA (cont.)

• Pros
• Develop internal expertise
• Internal staff now trained in project management
• Cons
• Can be disagreement on the interpretation of the
  standard
• The model and the vendor may use different
  terminology and life cycles

20
External Certification

• Using external standards
• Select appropriate standard
• Require certification from vendors in standard
• Make certain that you understand the
  certification process
• All models assume a specific customer behavior

21
External Certification (cont.)

• Pros
• Organizations must be certified to compete in
  procurement
• Certification costs are covered by the vendor
• Cons
• Very few health and human services vendors have
  any type of certification
• Would need to be national to be effective

22
Process Verification

• Are the vendors activities following these
  defined processes?
• Constant review of vendors activities and
  products to ensure compliance
• Requires an internal presence by QA personnel
• Can be supported by vendors QA group

23
Product Quality Assurance

• Sometimes called IVV (Independent Verification
  and Validation)
• Reviewing, testing, and using the delivered
  product to verify and validate the product

24
Estimating Cost of External QA

• Guideline
• Use 5 of the total cost of the project computed
  after all other costs are identified
• Example
• A project with development contract of 1.5
  million should budget no less than 75,000 for
  their QA vendor

25
Summary

• QA needs to be based on a standard
• QA vendors should specify the standard that they
  use
• Internal QA requires a trained staff
• What experiences have you had with QA?