8130 and 4130

1
8130 and 4130
Test Design Documentation Adrian Marshall
2
Test Design

• Logical grouping of tests
• Test size
• Modular test design
• Test class checklists
• Prioritisation of tests
• Test coverage

3
Test Design

• Group Tests by
• Test classes
• White, black, grey box
• Focus areas
• Functionality
• Usability
• Reliability
• Performance
• D, I, A, T (demo, inspection, analysis, test)

4
Test Design

• Requirements testing
• Group by
• Similar functions
• Similar set-up conditions
• Minimise verification of a requirement across
  multiple tests / test cases
• Manage requirements coverage (and traceability)
  in the design

5
Test Design

• Size of tests
• Aim to have self-contained tests that can be
  executed within an hour
• Where possible have pre-established test data
• Ensure tests have set-up and re-set conditions at
  the end of the test
• Use (develop) test class checklists for test
  design guidance

6
Test Design

• Prioritisation of tests
• Functional criticality
• Build sequence
• Test coverage
• Criticality
• Complexity
• Product history

7
Test Design

• Test class checklists
• Knowledge base of verification
• Environment characteristics
• Functional focus
• Identifies testing at various levels

8
Test Design

• In small groups develop some checklists for the
  following test classes
• HMI (Human Machine Interface) functionality
• COTS (Commercial off the Shelf) software
• Security (encryption, authentication, control)

9
Test Documentation

• Why do we document?
• Communication
• Repeatability
• Records of what needs to be done and what was
  done
• To manage test coverage

10
IEEE Test Documentation Set

• IEEE 829/ AS 4006
• Planning preparation
• Test Plans
• Test Design Specifications
• Test Case Specifications
• Test Procedure Specifications
• Execution reporting
• Test Item Transmittal Report
• Test Log
• Test Incident Report
• Test Summary Report
• Other useful documents
• Test Verification Matrix
• Test Activity Measurement Report.

11
Test Documentation
Other terms for test documents Test
Specifications Test Descriptions
12
Test Documentation

• Test Plans
• Master Test Plan
• Define testing at all levels
• Avoid needless overlaps and dangerous gaps
• Defines subordinate test plans
• Consists of test concept and summary test plan
• Can be delivered in multiple phases

13
Test Documentation

• Other Test Plans
• Plan tests for
• sub-systems,
• major functional groups, and/or
• test harnesses

14
Test Documentation

• Test Design / Test Case specification
• Defines test design and major test cases
• Describes how the tests are to be run
• Details the test environment
• Describes the test cases (not to a procedural
  level), aims and objectives

15
Test Documentation

• Test Design and Case specification
• 1 INTRODUCTION 3
• 2 REFERENCES 3
• 3 TEST APPROACH 3
• 3.1 Test design 3
• 3.2 Test Conduct 3
• 3.2.1 Test Readiness 3
• 3.2.2 Test Execution Strategy 3
• 3.2.3 Pass / Fail determination 3
• 3.3 Test Reporting 3

16
Test Documentation

• Test Case spec (contd)
• 4 TEST ENVIRONMENT 3
• 5 REQUIREMENTS 3
• 6 TEST DESCRIPTION 3
• 6.1 Test Name 1
• 6.2 Test Name 2
• 7 DEFINITIONS, ABBREVIATIONS AND ACRONYMS 3

17
Test Documentation

• Test Case spec (contd)
• APPENDIX A HARDWARE/SOFTWARE CONFIGURATIONS 3
• APPENDIX B SOFTWARE SETUP 3
• APPENDIX C TEST DATA 3

18
Test Documentation

• Test Procedures
• Include in Test Case (or have as a separate
  document)
• Can have delivery in multiple phases
• Describes the test steps
• Can be very prescriptive (beware of maintenance
  of such procedures)
• High level procedures may not be very repeatable
  (good and bad with this)

19
Test Documentation

• Test Logs / test record
• Identify actual test schedule and test duration
• Describe software and hardware versions
• Identify any deviations from the test procedures
• List who conducted the tests and any witnesses
  (eg customer, QA)
• Can be included in the case/ procedure document

20
Test Documentation

• Incident/defect reports
• Describe the issues raised
• To include
• Title
• Adequate description of the issue
• Test reference
• Configuration info (software/ hardware versions,
  test data)

21
Test Documentation

• Incident/defect reports
• To include
• Priority / severity information
• Who assigned to
• Status of the issue (open, under investigation,
  being fixed, fixed, closed etc.
• Dates (open, closed)

22
Test Documentation

• Incident/defect reports
• Should not have restrictions on creation
• Should only be closed the raiser or delegate

23
Test Documentation

• Test Reports
• Identify
• What was tested
• When
• By whom
• On what
• Outcomes (pass / fail)
• Issues (reference defect number)

24
Test Documentation

• Test Reports
• Written on conclusion of test
• Aimed to report upwards
• Report recommendations should be documented
  separately

25
Test Documentation

• Test Verification Matrix
• Maps the test
• Plans/designs
• cases
• procedures
• to the project requirements to ensure that
  all testable requirements are verified and to
  manage requirement changes.

26
Test Documentation

• Activity Measurement Report
• Quantifies the testing activities and their
  results
• e.g. hours per activity
• start and end dates
• number of defects detected
• defect trends
• analysis of testing activities against plans.

27
Summary

• Design
• Define methods, priorities, logical groupings,
  size of tests, test coverage, and test
  environment / tools
• Use / develop checklists
• Documentation
• Pre execution - Plans, cases, procedures
• Execution logs
• Post execution reports
• Documents communicate, provide repeatability, aid
  in management of test coverage (needless overlaps
  and dangerous gaps)
• Aim for the document is to support test execution
  not be a distraction to it.