Software Testing Introduction

1
Software TestingIntroduction
2
Introduction

• An introduction to software testing.
• Serves as a tutorial to formal testing of
  software
• Topics covered
• definitions of testing
• validation and verification
• levels of testing from unit testing through to
  acceptance testing
• relationship with requirements and design
  specifications and
• test documentation.

3
What is Software Testing?

• Many definitions of software testing
• All definitions boil down to
• software testing is the process of executing
  software in a controlled manner, in order to
  answer the question "Does the software behave as
  specified?".
• Software testing is often used in association
  with the terms verification and validation.
• Verification is the checking or testing of items,
  including software, for conformance and
  consistency with an associated specification.
• Software testing is just one kind of
  verification, which also uses techniques such as
  reviews, analysis, inspections and walkthroughs.

4
What is Software Testing?

• Validation is the process of checking that what
  has been specified is what the user actually
  wanted.
• Validation Are we doing the right job?
• Verification Are we doing the job right?
• bug refers to a problem or fault in a computer.
  
• Software testing should not be confused with
  debugging.
• Debugging is the process of analyzing and
  locating bugs when software does not behave as
  expected.
• A methodical approach to software testing is a
  much more thorough means of identifying bugs.

5
What is Software Testing?

• Debugging supports testing, but cannot replace
  testing.
• However, no amount of testing can guarantee
  discovery of all bugs.
• Other activities often associated with software
  testing are
• static analysis
• dynamic analysis
• Static analysis investigates the source code,
  looking for problems and gathering metrics
  without executing the code.
• Dynamic analysis examines the behavior of
  executing software.

6
The Cost of Testing

• It is well known that most of an iceberg is
  hidden under water.
• The same is true of software.
• The number of lines of code in the product often
  describes the size of a software project, but
  that's just the tip of the iceberg.
• The amount of software that needs to be written
  to test the product can be staggering.
• Often the number of lines of code of test
  software will exceed the lines of code in the
  product software.

7
Cost

• Suppose you have to write three lines of test
  software for every new line of software you
  deliver.
• Then suppose management says that all your test
  software must be fully tested.
• So for every line of test software you write, you
  need to write 3 lines of software that tests the
  software that tests your product.
• If X is the number of lines in your product, you
  will have to write 3X lines of test software, and
  9X lines of code that tests the test software.
• Your job has just increased by a factor of 12!

8
Cost

• What generally happens is that people write as
  much test software as they can in the time left
  over at the end of the project.
• The test software is inadequate, so the product
  gets shipped with some bugs in it.
• Testing is a big job.
• You have to plan money, people, and most of all,
  TIME for it.
• Don't expect to write and debug all your test
  code in the last month before the critical design
  review.

9
Cost

• Even if you could, it wouldn't do you much good.
• By that time the design is cast in concrete, so
  nobody is going to change anything unless your
  test programs find catastrophic errors.
• If you find moderate errors, people will say,
  "That's a shame, but we can't do anything about
  it now. Why didn't you catch these errors
  sooner?"
• You have to code and test early in the
  development cycle while there is still time to
  take corrective action.

10
Software Specifications and Testing

• Validation and verification activities, such as
  software testing, cannot be meaningful unless
  there is a specification for the software.
• Depending on the size of the development,
  specification of software can range from a single
  document to a complex hierarchy of documents.
• A hierarchy of software specifications will
  typically contain three or more levels of
  software specification documents.
• Requirements Specifications specify what the
  software is required to do and may also specify
  constraints on how this may be achieved.

11
Software Specifications and Testing

• Architectural Design Specifications describe the
  architecture of a design which implements the
  requirements. Components within the software and
  the relationship between them.
• Detailed Design Specifications describe how each
  component in the software, down to individual
  units, is to be implemented.

12
Software Specifications and Testing
13
Software Specifications and Testing

• A hierarchy of specifications, makes possible
  testing software at various stages of the
  development.
• The levels of testing which correspond to the
  hierarchy of software specifications
• Unit Testing each unit (basic component) of the
  software is tested
• Integration Testing progressively larger groups
  of tested software components corresponding to
  elements of the architectural design are
  integrated and tested until the software works as
  a whole.
• System Testing, the software is integrated to the
  overall product and tested to show that all
  requirements are met.

14
Software Specifications and Testing

• Acceptance Testing upon which acceptance of the
  completed software is based. This will often use
  a subset of the system tests, witnessed by the
  customers for the software or system.
• Once each level of software specification has
  been written, the next step is to design the
  tests.
• Tests should be designed before the software is
  implemented.
• if the software was implemented first, it would
  be tempting to test the software against what it
  is observed to do rather than against what it is
  specified to do.

15
Software Specifications and Testing

• Within each level of testing, test results are
  evaluated.
• If a problem is encountered, either the tests are
  revised and reapplied, or the software is fixed
  and the tests reapplied.
• This is repeated until no problems are
  encountered, at which point development can
  proceed to the next level of testing.
• Testing does not end at the conclusion of
  acceptance testing.
• Software has to be maintained to fix problems
  which show up during use and to accommodate new
  requirements.

16
Software Specifications and Testing

• Software tests have to be repeated, modified and
  extended.
• Revision and repetition of tests forms a major
  part of the overall cost of developing and
  maintaining software.
• The term regression testing is used to refer to
  the repetition of earlier successful tests in
  order to make sure that changes to the software
  have not introduced side effects.

17
Test Design Documentation

• Tests design is subject to same engineering
  principles as the design of software.
• Good design consists of a number of stages which
  progressively elaborate the design of tests from
  a high level strategy to detailed test
  procedures.
• These stages are
• test strategy,
• test planning,
• test case design, and
• test procedure design.

18
Test Design Documentation

• The design of tests is driven by the
  specification of the software.
• At the highest level tests will be designed to
  verify that the software faithfully implements
  the Requirements Specification.
• At lower levels tests will be designed to verify
  that items of software implement all design
  decisions made in the Architectural Design
  Specification and Detailed Design Specifications.
  
• As with any design process, each stage of the
  test design process should be subject to informal
  and formal review.

19
Test Strategy

• A test strategy is a statement of the overall
  approach to testing, identifying what levels of
  testing are to be applied and the methods,
  techniques and tools to be used.
• A test strategy should be applicable to all of an
  organizations software developments.
• Developing a test strategy which efficiently
  meets the needs of an organization is critical to
  the success of software development within the
  organization.
• The application of a test strategy to a software
  development project should be detailed in the
  projects software quality plan.

20
Test Plans

• The next stage of test design is development of a
  test plan.
• A test plan states the
• items to be tested are,
• at what level they will be tested,
• what sequence they are to be tested in,
• how the test strategy will be applied to the
  testing of each item,
• and describes the test environment.
• A test plan may be project wide, or a hierarchy
  of plans relating to the various levels of
  specification and testing

21
Plans

• An Acceptance Test Plan describes the plan for
  acceptance testing of the software. Usually
  published as a separate document, but might be
  published with the system test plan.
• A System Test Plan describes the plan for system
  integration and testing. Usually published as a
  separate document, but might be published with
  the acceptance test plan.
• A Software Integration Test Plan describing the
  plan for integration of tested software
  components. May form part of the Architectural
  Design Specification.
• Unit Test Plan(s) describes the plans for
  testing of individual units of software. May form
  part of the Detailed Design Specifications.

22
Plans

• Each test plan provides a plan for verification
  that the software fulfills the requirements or
  design statements of the software specification.
• In the case of acceptance testing and system
  testing, this means the Requirements
  Specification.

23
Test Plans

• Test Plan Outline (Adapted from IEEE Standard for
  Software Test Documentation (829) )
  
  
• 1.BACKGROUND
• 2.INTRODUCTION
• 3.ASSUMPTIONS
• 4.TEST ITEMS
• List each of the items (programs) to be
  tested

24
Plan

• 5.FEATURES TO BE TESTED
• List each of the features (functions or
  requirements) which will be tested or
  demonstrated by the test.
• 6.FEATURES NOT TO BE TESTED
• Explicitly lists each feature, function, or
  requirement which won't be tested and why not.
• 7.APPROACH
• Describe the data flows and test philosophy.
• Simulation or Live execution, Etc.
• 8.ITEM PASS/FAIL CRITERIA Blanket statement
• Itemized list of expected output and
  tolerances

25
Plan

• 9.SUSPENSION/RESUMPTION CRITERIA
• Must the test run from start to completion?
• Under what circumstances may it be resumed
  in the middle?
• Establish check-points in long tests.
• 10.TEST DELIVERABLES
• What, besides software, will be delivered?
• Test report
• Test software
• 11.TESTING TASKS Functional tasks (e.g.,
  equipment set up)
• Administrative tasks

26
Plan

• 12.ENVIRONMENTAL NEEDS
• Security clearance
• Office space equipment
• Hardware/software requirements
• 13.RESPONSIBILITIES
• Who does the tasks in Section 10?
• What does the user do?
• 14.STAFFING TRAINING
• 15.SCHEDULE
• 16.RESOURCES
• 17.RISKS CONTINGENCIES
• 18.APPROVALS

27
Test Case Design

• After the test plan stage, the next stage of test
  design is to specify a set of test cases for each
  item to be tested at that level.
• A number of test cases will be identified for
  each item to be tested at each level of testing.
• Each test case specifies how the implementation
  of a particular requirement or design decision is
  to be tested and the criteria for success of the
  test.
• The test cases may be documented with the test
  plan, as a section of a software specification,
  or in a separate document called a test
  specification or test description.

28
Case Design

• The specification of the following test cases may
  be documented by a separate document or in the
  appropriate test plan
• acceptance testing
• system integration
• for each stage of integration of tested software
  components
• testing of individual units of software
• System testing and acceptance testing involve an
  enormous number of individual test cases.
• an index which cross references between
  requirements and test cases. Referred to as a
  Verification Cross Reference Index (VCRI) and is
  attached to the test specification.

29
Case Design

• Cross reference indexes may also be used with
  unit testing and software integration testing.
• Design test cases for both positive testing and
  negative testing.
• Positive testing checks that the software does
  what it should.
• Negative testing checks that the software doesn't
  do what it shouldn't.

30
Test Procedures

• The final stage of test design implements a set
  of test cases as a test procedure, specifying the
  exact process to be followed to conduct each of
  the test cases.
• For each item to be tested, at each level of
  testing, a test procedure specifies the process
  to be followed in conducting the appropriate test
  cases.
• A test procedure cannot leave out steps or make
  assumptions. The level of detail must be such
  that the test procedure is deterministic and
  repeatable.
• Test procedures should always be separate items,
  because they contain a great deal of detail which
  is irrelevant to software specifications.

31
Results Documentation

• The outputs of each test execution should be
  recorded.
• Results are assessed against criteria in the test
  specification
• Each test execution should also be recorded in a
  test log.
• The test log contains records of
• when each test run,
• outcome of each test execution, and
• may include key observations made during test
  execution.
• A log may not be exist for unit and software
  integration tests.
• Test reports may be produced during the testing
  process.
• A test report summarizes the results and
  documents analysis.
• An acceptance test report often is a contractual
  document.

32
Summary

• List of rules to an aid effective software
  testing.
• Always test against a specification.
• Document the testing process specify tests and
  record test results.
• Test hierarchically against each level of
  specification. Finding more errors earlier will
  ultimately reduce costs.
• Plan verification and validation activities,
  particularly testing.
• Complement testing with techniques such as static
  analysis.
• Test positively that the software does what it
  should, but also negatively that it doesn't do
  what it shouldn't.