Testing Strategies

1
Testing Strategies

• TECH2001
• Jon Ivins

2
Overview

• What is testing?
• Is evaluation the same thing?
• Different levels of testing..
• and when to do them

3
Testing V Evaluation

• Testing is the process of matching a prescribed
  set of quantifiable criteria against performance
  to find errors
• Evaluation looks for improvements to the design
  during development or at completion
• Both quotes from England and Finney

4
Testing Methods

• Concept Testing
• Try out main project ideas on a subset of target
  audience
• Usually based upon a discussion

Pros Quick to set up, relatively cheap, initial
concepts can be adjusted quickly
Cons Selecting the focus group depends upon
correct marketing information
5
Testing Methods

• Prototyping
• Mimics ideas for final design but with limited
  functionality

Pros Simple to do, relatively inexpensive, quick
to do, allows active user involvement with design
process
Cons Not all functions discussed so tasks might
be missed from final design, focus on interface
rather than task constraints
6
Testing Methods

• Peer review
• Utilises experienced colleagues to refine design
  during early stages

Pros Can gain valuable feedback
Cons Dependant on level of experience AND what
is know about the project goals at the time
7
Testing Methods

• Module Testing
• Each software module is tested for errors
• Common tests include data typing, range and
  limit checks, navigation, accessing other
  programs, storing data, etc.

Pros Conceptually simple, forced to define test
criteria during design stage
Cons Time consuming, all errors cannot be found
8
Testing Methods

• System Testing
• When all modules have been written and tested,
  application assembled and tested

Pros Gives confidence, NOT certainty, that
application will work when released, most errors
trapped
Cons Time consuming but does not guarantee
bug-free code, users actions cannot always be
predicted
9
Testing Methods

• Usability Tests
• Users test the application during development and
  before release

Pros Can find unforeseen user behaviour
Cons requires special labs to observe users,
time consuming, costly, huge amounts of data
generated
10
Testing Methods

• Field Trials
• In-situ with users

Pros Best chance of detecting errors in design
or coding before going live
Cons staff have to visit client (cost/time),
dependent upon observational skills, huge amounts
of data generated
11
Testing Methods

• Acceptance Testing
• Does the application meet the specified
  requirements of the client?

Pros Once complete, the project is almost over!
Cons Clients always change their mind / request
changes!
12
Towards a testing strategy

• Always agree functionality , usability and
  performance requirements with the client BEFORE
  starting work.
• Always agree test criteria for the above BEFORE
  starting work
• Get the test criteria in writing as part of the
  contract BEFORE starting work

13
Development Testing

• Happens throughout development
• Can use any form of testing
• All stages of testing must be documented to
  identify faults, action taken, knock-on effects
  etc. See Prince II for guidelines.
• A proper change control mechanism must be in
  place
• All stages must be signed off as completed

14
Acceptance Testing

• Usually happens at the end
• Usually in-situ testing is involved
• Based upon criteria in testing strategy
• Client may carry out their own system tests
• Must be logged and acceptance filed

15
Final sign-off

• Client, users are satisfied that application is a
  specified.
• This is when you get paid!

16
Summary

• Multimedia does not have a strong tradition of
  testing. It has not developed a clear
  relationship with the methods of testing from
  other disciplines and has not devised its own.
• There are many forms of testing. Each has its
  strengths and weaknesses in relation to the needs
  of an application.
• The project manager has to balance the needs of
  the application with time and cost factors to
  devise a testing strategy.
• The clients have to agree to the strategy in the
  full knowledge of what it includes and what it
  excludes so there are no last minute
  recriminations.

17
Developmental Testing is the combination of
sign-offs of the completed parts of the project
as well as the natural refinement of the
individual assets during the development
process.Acceptance Testing takes place at the
final stage of the project to pre-defined
criteria specified in the testing strategy. The
integrated code is tested and the robustness of
the project is scrutinised.Final sign-off agrees
that the team has met its responsibilities and
the application performs as specified.
18
References / Further Reading
Managing Multimedia project Management for
Interactive Media 2nd ed by England and Finney,
Addison Wesley 1999