Dimensions of Testing - PowerPoint PPT Presentation

Loading...

PPT – Dimensions of Testing PowerPoint presentation | free to view - id: 24ab6a-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Dimensions of Testing

Description:

Development process normally includes unit testing by developers ... Only two are considered: scale. skills. roadmap for software test engineering: ... – PowerPoint PPT presentation

Number of Views:19
Avg rating:3.0/5.0
Slides: 46
Provided by: dian183
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Dimensions of Testing


1
Dimensions of Testing 1
  • CFICSE
  • October 2002
  • Diane Kelly Terry Shepard

2
Dimensions of Testing Part 1
3
Why Dimensions of Testing?
  • Testing tends to be viewed as an activity that
    testers do
  • implicit is the thought that all testers have
    more or less the same skill set
  • body of knowledge is too big for this to be true
  • categorization helps to plan resource use for
    testing
  • also a basis for training and education
  • Help make better informed decisions

4
Outline Part 1
  • Definitions of testing
  • Testing is hard!
  • Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

5
Definitions of Testing (1)
  • 15 Myers
  • The Process of executing a program with the
    intent of finding errors
  • 10 Hetzel
  • "Testing is any activity aimed at evaluating an
    attribute or capability of a program or system
    and determining that it meets its required
    results."

6
Definitions of Testing (2)
  • IEEE Standard 610.12-1990 30
  • The process of operating a system or component
    under specified conditions, observing or
    recording the results, and making an evaluation
    of some aspect of the system or component
  • IEEE Standard 829-1998 18
  • The process of analyzing a software item to
    detect the differences between existing and
    required conditions (that is, bugs) and to
    evaluate the features of the software item.

7
Not Testing (IEEE 610.12 30)
  • Static Analysis - e.g.
  • proof of correctness
  • style checking
  • Dynamic Analysis that does not require execution
    of the code e.g.
  • reachability analysis dead code
  • checks for memory leaks in C

8
Secondary Definitions
  • finding
  • issue identified in a product by inspection
  • defect
  • a finding that needs corrective action
  • fault
  • a defect that may cause a failure
  • failure
  • a behaviour of executing code that a tester
    thinks is wrong
  • application of these terms requires judgment
  • decisions vary, depending on the situation and on
    individual background
  • disagreements are common!

9
Testing is Hard!
  • Case is made by James Whittaker in 31
  • hits budget and schedule limits
  • may include heavy debugging
  • may even include product (re-)development
  • bugs slip through to customers
  • Testing evaluates more than just correctness

10
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

11
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

12
 Testing Process Overview
  • What are the Testing Activities?
  • When are they performed?
  • Stages of Testing
  • Regression testing is part of every stage
  • Parts to be Tested
  • Effects of Development Process
  • Testing process diversity
  • Examples and Issues
  • Place of Debugging in the Testing Process

13
 Testing Activities
  • Test planning and design
  • Verification of test plans, designs and cases
  • Test case selection, creation, and verification
  • Planning expected results oracles, …
  • Data preparation
  • Execution and recording of results
  • Analysis of results
  • Coverage measurement
  • Postprocessing volume reduction, …
  • Wrap-up

14
Stages of Testing
  • The testing process must organize the stages of
    testing, which are essentially determined by the
    parts to be tested e.g.
  • unit
  • increment
  • build
  • integration
  • system
  • release
  • installation
  • user interface

15
Parts to be Tested
  • Any part of a system can be tested
  • choice of part constrains choices in other
    dimensions
  • Units are typically the smallest parts tested
  • a unit can be a function, a class, or a work
    assignment
  • Some parts of the software may not be tested at
    all
  • e.g. exception handlers, debug code, commercial
    components, library functions
  • For some testing
  • stubs and drivers may have to be written (and
    tested)
  • PUT Part Under Test

16
Effects of Development Process on Testing Process
(1)
  • Impact of choice of development process e.g
  • waterfall
  • incremental
  • XP
  • safety critical …
  • Impact of development process implementation
    choices e.g.
  • daily builds
  • metrics
  • tool support (e.g. RUP Rational Unified Process)
  • change and configuration management, …
  • Impact of design approach e.g. OO vs. procedural

17
Effects of Development Process on Testing Process
(2)
  • Development process normally includes unit
    testing by developers
  • developers often have limited time and expertise
    for unit testing
  • expected quality and actual quality of units have
    major impact on the testing process
  • Testers must protect themselves
  • in normal cases, testers should talk to
    developers as early as possible in the process
  • in the worst case, testers become developers

18
Effects of Development Process on Testing Process
(3)
  • Resource limitations can be severe during testing
  • load, performance and platform testing during
    system testing are especially demanding
  • Limitations include one or more of
  • Budget
  • Schedule
  • Hardware
  • Testers
  • Debuggers
  • Fixers
  • Users

19
Testing Process Diversity Examples and Some
Issues
  • Examples of decisions that affect process
  • Microsoft testing buddies
  • test then code (XP)
  • operational analysis (reliability testing)
  • …
  • Some issues that must be considered in making
    testing process decisions
  • defect tracking style
  • inspection choices e.g. test designs, plans,
    cases,
  • geographic distribution of a project
  • …

20
Debugging in the Testing Process
  • Debugging is a separate activity from testing
  • It has a special place in the testing process
  • testing reveals the need to do debugging
  • fine grain testing is often used in debugging
  • depends on code review to choose additional tests
  • debugging and testing are usually interwoven
  • e.g. in defect tracking systems, defects are
    logged as a result of testing, and debugging
    supplies more information about each defect

21
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

22
Qualities (Ilities)
  • All the ilities can affect testing, but some have
    a special relationship to testing e.g.
  • testability, esp. observability and
    controllability
  • failure rate testing stops if it is too high
  • Assessing a property (ility) of a PUT may be a
    purpose of testing e.g. correctness, reliability,
    usability, performance, …
  • may affect choices of strategies or techniques
  • only appropriate at certain stages
  • assessment of some ilities, such as reliability
    or performance, is difficult without testing
    automation

23
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose of testing
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

24
 Context/environment (1)
  • Circumstances in which a part is tested
  • Testing in the development environment
  • simulation
  • Testing in the deployment environment(s)
  • actual CPU, OS, devices, …
  • There may be version and release issues
  • Languages of expression
  • scripting languages OS shell scripts, tcl, perl,
    …
  • of the program
  • language version, compiler, …
  • of the test implementation
  • can be a proprietary language e.g. SQA Basic for
    Rational Robot
  • Operating system/hardware
  • MS windows, Risc UNIX, Mac O/S, Vax VMS
  • will affect availability of tools

25
 Context/environment (2)
  • Application domain, e.g.
  • military weapons systems
  • household appliances
  • military logistics
  • financial
  • scientific, …
  • Shrink wrapped vs. single customer
  • Distributed or not
  • e.g. Internet or web based
  • Part of a product family or not
  • Organizational context
  • e.g. software architecture may match the
    architecture of the organization
  • Role of standards in testing
  • …

26
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose of testing
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

27
 Purposes of testing ...
  • Why is the testing being done?
  • find failures (not bugs!)
  • certification
  • safety criticality
  • usability
  • performance
  • acceptance
  • load
  • porting
  • compatibility
  • security

28
 More purposes of testing...
  • Why is the testing being done?
  • fault tolerance and recovery
  • measure reliability
  • estimate defects remaining
  • decide when to release
  • assess problems in software development process
  • confirm that functionality is not broken
    following modification
  • all the rest of the ilities!
  • …

29
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose of testing
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

30
Definitions Strategy, Techniques, …
  • Strategy outlines in broad terms how to use
    testing to assess the extent to which one or more
    goals for a product have been met
  • Techniques implement strategies, provide the
    data needed to do the assessment
  • Some people use other terminology
  • Test planning and design captures the choices
    of strategy and techniques
  • Tactics a narrower version of strategy?
  • Approach some mixture of the above?
  • …

31
 Strategy/techniques
  • Both strategy and techniques guide how test cases
    are chosen
  • Distinction between strategy and technique is
    soft
  • Next two slides contain some examples
  • some of the examples could be on either slide

32
Examples of Strategies
  • Exploratory testing
  • Requirements based test design
  • Design based testing
  • Testing in the small
  • Testing in the large
  • More to come …

33
Examples of Strategies that are close to
Techniques
  • alpha/beta testing
  • testing most used part of the code vs. buggiest
    part
  • tactic?
  • testing GUI vs. testing functionality
  • design testing (requires an executable design)
  • laboratory testing usability, …
  • operational profiles
  • specification based testing

34
Examples of Techniques
  • Ad hoc
  • Partitions boundaries, intervals, …
  • Data flow
  • Combinational decision tables, …
  • State based mode transition, …
  • Test Scenarios
  • Guerilla testing
  • smoke tests
  • Fault injection, mutation testing
  • White box, black box (interface testing), gray
    box
  • more to come …

35
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose of testing
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

36
Automation
  • What aspects of testing can be automated?
  • Coverage tools
  • Data generation
  • Capture/playback
  • Setup, clean-up, reinitialization
  • Push-button execution of test cases
  • Data filters
  • Post analysis of output
  • Configuration management
  • Oracles
  • Simulation of the environment
  • Recovery and restart
  • more to come …

37
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose of testing
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

38
 Adequacy
  • How is the decision to stop testing made?
  • Coverage
  • Limits set by mgmt
  • User acceptance e.g. Early adopters
  • Contractual
  • Reliability threshold
  • Fault detection rate threshold
  • Estimates of bugs left
  • Meeting a standard e.g. mc/dc
  • run out of time and money
  • more to come …

39
Dimensions of Testing
  • Process
  • Qualities
  • Context/environment
  • Purpose
  • Strategy/techniques
  • Automation
  • Adequacy
  • Cross cutting issues

40
 Cross cutting issues
  • These are issues that have an impact on all the
    other dimensions of testing
  • Only two are considered
  • scale
  • skills
  • roadmap for software test engineering
  • http//www.sdmagazine.com/articles/2001/0102/0102b
    /0102b.htm

41
Examples of Scale Issues
  • Size of project staff complement
  • communication issues
  • Size of customer base
  • determining customer usage patterns
  • Size/complexity of code (KLOC/McCabe)
  • adequacy of testing
  • Size of data sets / database
  • sufficient regression testing
  • Length of runtime
  • turnaround time for tests
  • Extent of output data
  • finding errors in output

42
Skills of a Tester (1)
  • Not all testers are developers
  • e.g. testers at Corel who are photographers
    (Corel Draw testing - end user testing)
  • e.g. testing buddies at Microsoft
  • Testers may also be developers
  • Testers must have
  • ability to select good test cases
  • ability to withstand the most severe deadline and
    resource pressures
  • ability to organize and analyze large volumes of
    data
  • ability to sort out complex version control issues

43
Skills of a Tester (2)
  • Inexperienced testers often fail to report
  • timing problems
  • transient bugs
  • what they cant quickly replicate
  • what they think can be attributable to their
    misunderstanding
  • what they think the reader of the report might
    consider minor
  • lots of other problems
  • Kaner et al 16

44
Roadmap for Software Test Engineering
(sdmagazine, Feb01)
  • Software engineering skills
  • understanding the rules of software engineering
  • computer programming
  • operating system level knowledge
  • test development must be done with same care as
    application development
  • communication skills
  • organizational skills
  • hands-on experience
  • attitude
  • destructive creativity
  • detective skills
  • understanding the product as the sum of its parts
  • appreciating the customers perspective
  • requirements change
  • skeptical but not hostile attitude
  • ability to be the bearer of bad news and remain
    objective
  • an eagerness to embrace new technologies

45
More Details
  • Strategies and Techniques
  • Regression Testing
  • Adequacy
  • Automation
About PowerShow.com