Testing the System - PowerPoint PPT Presentation

About This Presentation
Title:

Testing the System

Description:

Chapter 9. Testing the System. Shari L. Pfleeger. Joann M. Atlee. 4th Edition. 4th Edition ... Pfleeger and Atlee, Software Engineering: Theory and Practice ... – PowerPoint PPT presentation

Number of Views:73
Avg rating:3.0/5.0
Slides: 79
Provided by: hs91402
Learn more at: http://www.cs.ucf.edu
Category:
Tags: joann | system | testing

less

Transcript and Presenter's Notes

Title: Testing the System


1
Chapter 9
  • Testing the System
  • Shari L. Pfleeger
  • Joann M. Atlee
  • 4th Edition
  • 4th Edition

2
Contents
  • 9.1 Principles of system testing
  • 9.2 Function testing
  • 9.3 Performance testing
  • 9.4 Reliability, availability, and
    maintainability
  • 9.5 Acceptance testing
  • 9.6 Installation testing
  • 9.7 Automated system testing
  • 9.8 Test documentation
  • 9.9 Testing safety-critical systems
  • 9.10 Information systems example
  • 9.11 Real-time example
  • 9.12 What this chapter means for you

3
Chapter 9 Objectives
  • Function testing
  • Performance testing
  • Acceptance testing
  • Software reliability, availability, and
    maintainability
  • Installation testing
  • Test documentation
  • Testing safety-critical systems

4
9.1 Principles of System TestingSource of
Software Faults During Development
5
9.1 Principles of System Testing System Testing
Process
  • Function testing does the integrated system
    perform as promised by the requirements
    specification?
  • Performance testing are the non-functional
    requirements met?
  • Acceptance testing is the system what the
    customer expects?
  • Installation testing does the system run at the
    customer site(s)?

6
9.1 Principles of System Testing System Testing
Process (continued)
  • Pictorial representation of steps in testing
    process

7
9.1 Principles of System TestingTechniques Used
in System Testing
  • Build or integration plan
  • Regression testing
  • Configuration management
  • versions and releases
  • production system vs. development system
  • deltas, separate files and conditional
    compilation
  • change control

8
9.1 Principles of System TestingBuild or
Integration Plan
  • Define the subsystems (spins) to be tested
  • Describe how, where, when, and by whom the tests
    will be conducted

9
9.1 Principles of System TestingExample of Build
Plan for Telecommunication System
Spin Functions Test Start Test End
O Exchange 1 September 15 September
1 Area code 30 September 15 October
2 State/province/district 25 October 5 November
3 Country 10 November 20 November
4 International 1 December 15 December
10
9.1 Principles of System TestingExample Number
of Spins for Star Network
  • Spin 0 test the central computers general
    functions
  • Spin 1 test the central computers
    message-translation function
  • Spin 2 test the central computers
    message-assimilation function
  • Spin 3 test each outlying computer in the stand
    alone mode
  • Spin 4 test the outlying computers
    message-sending function
  • Spin 5 test the central computers
    message-receiving function

11
9.1 Principles of System TestingRegression
Testing
  • Identifies new faults that may have been
    introduced as current one are being corrected
  • Verifies a new version or release still performs
    the same functions in the same manner as an older
    version or release

12
9.1 Principles of System TestingRegression
Testing Steps
  • Inserting the new code
  • Testing functions known to be affected by the new
    code
  • Testing essential function of m to verify that
    they still work properly
  • Continuing function testing m 1

13
9.1 Principles of System TestingSidebar 9.1 The
Consequences of Not Doing Regression Testing
  • A fault in software upgrade to the DMS-100
    telecom switch
  • 167,000 customers improperly billed 667,000

14
9.1 Principles of System TestingConfiguration
Management
  • Versions and releases
  • Production system vs. development system
  • Deltas, separate files and conditional
    compilation
  • Change control

15
9.1 Principles of System TestingSidebar 9.2
Deltas and Separate Files
  • The Source Code Control System (SCCS)
  • uses delta approach
  • allows multiple versions and releases
  • Ada Language System (ALS)
  • stores revision as separate, distinct files
  • freezes all versions and releases except for the
    current one

16
9.1 Principles of System TestingSidebar 9.3
Microsofts Build Control
  • The developer checks out a private copy
  • The developer modifies the private copy
  • A private build with the new or changed features
    is tested
  • The code for the new or changed features is
    placed in master version
  • Regression test is performed

17
9.1 Principles of System TestingTest Team
  • Professional testers organize and run the tests
  • Analysts who created requirements
  • System designers understand the proposed
    solution
  • Configuration management specialists to help
    control fixes
  • Users to evaluate issues that arise

18
9.2 Function TestingPurpose and Roles
  • Compares the systems actual performance with its
    requirements
  • Develops test cases based on the requirements
    document

19
9.2 Function TestingCause-and-Effect Graph
  • A Boolean graph reflecting logical relationships
    between inputs (causes), and the outputs
    (effects) or transformations (effects)

20
9.2 Function TestingNotation for
Cause-and-Effect Graph
21
9.2 Function TestingCause-and-Effect Graphs
Example
  • INPUT The syntax of the function is LEVEL(A,B)
    where A is the height in meters of the water
    behind the dam, and B is the number of
    centimeters of rain in the last 24-hour period
  • PROCESSING The function calculates whether the
    water level is within a safe range, is too high,
    or is too low
  • OUTPUT The screen shows one of the following
    messages
  • 1. LEVEL SAFE when the result is safe or low
  • 2. LEVEL HIGH when the result is high
  • 3. INVALID SYNTAX
  • depending on the result of the calculation

22
9.2 Function TestingCause-and-Effect Graphs
Example (Continued)
  • Causes
  • The first five characters of the command LEVEL
  • The command contains exactly two parameters
    separated by a comma and enclosed in parentheses
  • The parameters A and B are real numbers such that
    the water level is calculated to be LOW
  • The parameters A and B are real numbers such that
    the water level is calculated to be SAFE
  • The parameters A and B are real numbers such that
    the water level is calculated to be HIGH

23
9.2 Function TestingCause-and-Effect Graphs
Example (Continued)
  • Effects
  • 1. The message LEVEL SAFE is displayed on the
    screen
  • 2. The message LEVEL HIGH is displayed on the
    screen
  • The message INVALID SYNTAX is printed out
  • Intermediate nodes
  • 1. The command is syntactically valid
  • 2. The operands are syntactically valid

24
9.2 Function TestingCause-and-Effect Graphs of
LEVEL Function Example
  • Exactly one of a set of conditions can be invoked
  • At most one of a set of conditions can be invoked
  • At least one of a set of condition can be invoked
  • One effects masks the observance of another
    effect
  • Invocation of one effect requires the invocation
    of another

25
9.2 Function TestingDecision Table for
Cause-and-Effect Graph of LEVEL Function
Test 1 Test 2 Test 3 Test 4 Test 5
Cause 1 I I I S I
Cause 2 I I I X S
Cause 3 I S S X X
Cause 4 S I S X X
Cause 5 S S I X X
Effect 1 P P A A A
Effect 2 A A P A A
Effect 3 A A A P P
26
9.2 Function TestingAdditional Notation for
Cause-and-Effect Graph
27
9.3 Performance TestsPurpose and Roles
  • Used to examine
  • the calculation
  • the speed of response
  • the accuracy of the result
  • the accessibility of the data
  • Designed and administrated by the test team

28
9.3 Performance TestsTypes of Performance Tests
  • Stress tests
  • Volume tests
  • Configuration tests
  • Compatibility tests
  • Regression tests
  • Security tests
  • Timing tests
  • Environmental tests
  • Quality tests
  • Recovery tests
  • Maintenance tests
  • Documentation tests
  • Human factors (usability) tests

29
9.4 Reliability, Availability, and
MaintainabilityDefinition
  • Software reliability operating without failure
    under given condition for a given time interval
  • Software availability operating successfully
    according to specification at a given point in
    time
  • Software maintainability for a given condition
    of use, a maintenance activity can be carried out
    within stated time interval, procedures and
    resources

30
9.4 Reliability, Availability, and
MaintainabilityDifferent Level of Failure
Severity
  • Catastrophic causes death or system loss
  • Critical causes severe injury or major system
    damage
  • Marginal causes minor injury or minor system
    damage
  • Minor causes no injury or system damage

31
9.4 Reliability, Availability, and
MaintainabilityFailure Data
  • Table of the execution time (in seconds) between
    successive failures of a command-and-control
    system

Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows) Interfailure Times (Read left to right, in rows)
3 30 113 81 115 9 2 91 112 15
138 50 77 24 108 88 670 120 26 114
325 55 242 68 422 180 10 1146 600 15
36 55 242 68 227 65 176 58 457 300
97 263 452 255 197 193 6 79 816 1351
148 21 233 134 357 193 236 31 369 748
0 232 330 365 1222 543 10 16 529 379
44 129 810 290 300 529 281 160 828 1011
445 296 1755 1064 1783 860 983 707 33 868
724 2323 2930 1461 843 12 261 1800 865 1435
30 143 108 0 3110 1247 943 700 875 245
729 1897 447 386 446 122 990 948 1082 22
75 482 5509 100 10 1071 371 790 6150 3321
1045 648 5485 1160 1864 4116
32
9.4 Reliability, Availability, and
MaintainabilityFailure Data (Continued)
  • Graph of failure data from previous table

33
9.4 Reliability, Availability, and
MaintainabilityUncertainty Inherent from Failure
Data
  • Type-1 uncertainty how the system will be used
  • Type-2 uncertainty lack of knowledge about the
    effect of fault removal

34
9.4 Reliability, Availability, and
MaintainabilityMeasuring Reliability,
Availability, and Maintainability
  • Mean time to failure (MTTF)
  • Mean time to repair (MTTR)
  • Mean time between failures (MTBF)
  • MTBF MTTF MTTR
  • Reliability R MTTF/(1MTTF)
  • Availability A MTBF (1MTBF)
  • Maintainability M 1/(1MTTR)

35
9.4 Reliability, Availability, and
MaintainabilityReliability Stability and Growth
  • Probability density function f or time t, f (t)
    when the software is likely to fail
  • Distribution function the probability of failure
  • F(t) ? f (t) dt
  • Reliability Function the probability that the
    software will function properly until time t
  • R(t) 1- F(t)

36
9.4 Reliability, Availability, and
MaintainabilityUniformity Density Function
  • Uniform in the interval from t0..86,400 because
    the function takes the same value in that interval

37
9.4 Reliability, Availability, and
MaintainabilitySidebar 9.4 Difference Between
Hardware and Software Reliability
  • Complex hardware fails when a component breaks
    and no longer functions as specified
  • Software faults can exist in a product for long
    time, activated only when certain conditions
    exist that transform the fault into a failure

38
9.4 Reliability, Availability, and
MaintainabilityReliability Prediction
  • Predicting next failure times from past history

39
9.4 Reliability, Availability, and
MaintainabilityElements of a Prediction System
  • A prediction model gives a complete probability
    specification of the stochastic process
  • An inference procedure for unknown parameters of
    the model based on values of t1, t2, , ti-1
  • A prediction procedure combines the model and
    inference procedure to make predictions about
    future failure behavior

40
9.4 Reliability, Availability, and
MaintainabilitySidebar 9.5 Motorolas
Zero-Failure Testing
  • The number of failures to time t is equal to
  • a e-b(t)
  • a and b are constant
  • Zero-failure test hour
  • ln ( failures/ (0.5 failures) X
    (hours-to-last-failure)
  • ln(0.5 failures)/(test-failures failures)

41
9.4 Reliability, Availability, and
MaintainabilityReliability Model
  • The Jelinski-Moranda model assumes
  • no type-2 uncertainty
  • corrections are perfect
  • fixing any fault contributes equally to improving
    the reliability
  • The Littlewood model
  • treats each corrected faults contribution to
    reliability as independent variable
  • uses two source of uncertainty

42
9.4 Reliability, Availability, and
MaintainabilitySuccessive Failure Times for
Jelinski-Moranda
I Mean Time to ith failure Simulated Time to ith Failure
1 22 11
2 24 41
3 26 13
4 28 4
5 30 30
6 33 77
7 37 11
8 42 64
9 48 54
10 56 34
11 67 183
12 83 83
13 111 17
14 167 190
15 333 436
43
9.5 Acceptance TestsPurpose and Roles
  • Enable the customers and users to determine if
    the built system meets their needs and
    expectations
  • Written, conducted and evaluated by the customers

44
9.5 Acceptance TestsTypes of Acceptance Tests
  • Pilot test install on experimental basis
  • Alpha test in-house test
  • Beta test customer pilot
  • Parallel testing new system operates in
    parallel with old system

45
9.4 Reliability, Availability, and
MaintainabilitySidebar 9.6 Inappropriate Use of
A Beta Version
  • Problem with the Pathfinders software
  • NASA used VxWorks operating system for PowerPCs
    version to the R6000 processor
  • A beta version
  • Not fully tested

46
9.4 Reliability, Availability, and
MaintainabilityResult of Acceptance Tests
  • List of requirements
  • are not satisfied
  • must be deleted
  • must be revised
  • must be added

47
9.6 Installation Testing
  • Before the testing
  • Configure the system
  • Attach proper number and kind of devices
  • Establish communication with other system
  • The testing
  • Regression tests to verify that the system has
    been installed properly and works

48
9.7 Automated System TestingSimulator
  • Presents to a system all the characteristics of a
    device or system without actually having the
    device or system available
  • Looks like other systems with which the test
    system must interface
  • Provides the necessary information for testing
    without duplication the entire other system

49
9.7 Automated System TestingSidebar 9.7
Automated Testing of A Motor Insurance Quotation
System
  • The system tracks 14 products on 10 insurance
    systems
  • The system needs large number of test cases
  • The testing process takes less than one week to
    complete by using automated testing

50
9.8 Test Documentation
  • Test plan describes system and plan for
    exercising all functions and characteristics
  • Test specification and evaluation details each
    test and defines criteria for evaluating each
    feature
  • Test description test data and procedures for
    each test
  • Test analysis report results of each test

51
9.8 Test DocumentationDocuments Produced During
Testing
52
9.8 Test DocumentationTest Plan
  • The plan begins by stating its objectives, which
    should
  • guide the management of testing
  • guide the technical effort required during
    testing
  • establish test planning and scheduling
  • explain the nature and extent of each test
  • explain how the test will completely evaluate
    system function and performance
  • document test input, specific test procedures,
    and expected outcomes

53
9.8 Test DocumentationParts of a Test Plan
54
9.8 Testing DocumentationTest-Requirement
Correspondence Chart
Test Requirement 2.4.1 Generate and Maintain Database Requirement 2.4.2 Selectively RetrieveData Requirement 2.4.3 Produced Specialized Reports
1. Add new record X
2. Add field X
3. Change field X
4. Delete record X
5. Delete field X
6. Create index X
Retrieve record with a requested
7. Cell number X
8. Water height X
9. Canopy height X
10. Ground cover X
11, Percolation rate X
12. Print full database X
13. Print directory X
14. Print keywords X
15. Print simulation summary X
55
9.8 Test Documentation Sidebar 9.8 Measuring
Test Effectiveness and Efficiency
  • Test effectiveness can be measured by dividing
    the number of faults found in a given test by the
    total number of faults found
  • Test efficiency is computed by dividing the
    number of faults found in testing by the effort
    needed to perform testing

56
9.8 Test DocumentationTest Description
  • Including
  • the means of control
  • the data
  • the procedures

57
9.8 Test DocumentationTest Description Example
  • INPUT DATA
  • Input data are to be provided by the LIST
    program. The program generates randomly a list
    of N words of alphanumeric characters each word
    is of length M. The program is invoked by
    calling
  • RUN LIST(N,M)
  • in your test driver. The output is placed in
    global data area LISTBUF. The test datasets to
    be used for this test are as follows
  • Case 1 Use LIST with N5, M5
  • Case 2 Use LIST with N10, M5
  • Case 3 Use LIST with N15, M5
  • Case 4 Use LIST with N50, M10
  • Case 5 Use LIST with N100, M10
  • Case 6 Use LIST with N150, M10
  • INPUT COMMANDS
  • The SORT routine is invoked by using the command
  • RUN SORT (INBUF,OUTBUF) or
  • RUN SORT (INBUF)
  • OUTPUT DATA
  • If two parameters are used, the sorted list is
    placed in OUTBUF. Otherwise, it is placed in
    INBUF.
  • SYSTEM MESSAGES
  • During the sorting process, the following message
    is displayed
  • Sorting ... please wait ...

58
9.8 Test DocumentationTest Script for Testing
The change field
  • Step N Press function key 4 Access data file.
  • Step N1 Screen will ask for the name of the
    date file.
  • Type systest.txt
  • Step N2 Menu will appear, reading
  • delete file
  • modify file
  • rename file
  • Place cursor next to modify file and press
    RETURN key.
  • Step N3 Screen will ask for record number.
    Type 4017.
  • Step N4 Screen will fill with data fields for
    record 4017
  • Record number 4017 X 0042 Y 0036
  • Soil type clay Percolation 4 mtrs/hr
  • Vegetation kudzu Canopy height 25 mtrs
  • Water table 12 mtrs Construct outhouse
  • Maintenance code 3T/4F/9R
  • Step N5 Press function key 9 modify
  • Step N6 Entries on screen will be highlighted.
    Move cursor to VEGETATION field. Type grass
    over kudzu and press RETURN key.
  • Step N7 Entries on screen will no longer be
    highlighted.
  • VEGETATION field should now read grass.

59
9.8 Test DocumentationTest Analysis Report
  • Documents the result of test
  • Provides information needed to duplicate the
    failure and to locate and fix the source of the
    problem
  • Provides information necessary to determine if
    the project is complete
  • Establish confidence in the systems performance

60
9.8 Test DocumentationProblem Report Forms
  • Location Where did the problem occur?
  • Timing When did it occur?
  • Symptom What was observed?
  • End result What were the consequences?
  • Mechanism How did it occur?
  • Cause Why did it occur?
  • Severity How much was the user or business
    affected?
  • Cost How much did it cost?

61
9.8 Test DocumentationExample of Actual Problem
Report Forms
62
9.8 Test DocumentationExample of Actual
Discrepancy Report Forms
63
9.9 Testing Safety-Critical Systems
  • Design diversity use different kinds of
    designs, designers
  • Software safety cases make explicit the ways
    the software addresses possible problems
  • failure modes and effects analysis
  • hazard and operability studies (HAZOPS)
  • Cleanroom certifying software with respect to
    the specification

64
9.9 Testing Safety-Critical SystemsUltra-High
Reliability Problem
  • Graph of failure data from a system in
    operational use

65
9.9 Testing Safety-Critical SystemsSidebar 9.9
Software Quality Practices at Baltimore Gas and
Electric
  • To ensure high reliability
  • checking the requirements definition thoroughly
  • performing quality reviews
  • testing carefully
  • documenting completely
  • performing thorough configuration control

66
9.9 Testing Safety-Critical SystemsSidebar 9.10
Suggestions for Building Safety-Critical Software
  • Recognize that testing cannot remove all faults
    or risks
  • Do not confuse safety, reliability and security
  • Tightly link the organizations software and
    safety organizations
  • Build and use a safety information system
  • Instill a management culture safety
  • Assume that every mistakes users can make will be
    made
  • Do not assume that low-probability, high-impacts
    event will not happen
  • Emphasize requirements definition, testing, code
    and specification reviews, and configuration
    control
  • Do not let short-term considerations overshadow
    long-term risks and cost

67
9.9 Testing Safety-Critical SystemsPerspective
for Safety Analysis
Known cause Unknown cause
Known effect Description of system behavior Deductive analysis, including fault tree analysis
Unknown effect Inductive analysis, including failures modes and effect analysis Exploratory analysis, including hazard and operability statistics
68
9.9 Testing Safety-Critical SystemsSidebar 9.11
Safety and the Therac-25
  • Atomic Energy of Canada Limited (AECL) performed
    a safety analysis
  • identify single fault using a failure modes and
    effects analysis
  • identify multiple failures and quantify the
    results by performing a fault tree analysis
  • perform detailed code inspections
  • AECL recommended
  • 10 changes to the Therac-25 hardware, including
    interlocks to back up software control energy
    selection and electron-beam scanning

69
9.9 Testing Safety-Critical SystemsHAZOP Guide
Words
Guide word Meaning
No No data or control signal sent or received
More Data volume is too high or fast
Less Data volume is too low or slow
Part of Data or control signal is incomplete
Other than Data or control signal has additional component
Early Signal arrives too early for system clock
Late Signal arrives too late for system clock
Before Signal arrives earlier in sequence than expected
After Signal arrives later in sequence than expected
70
9.9 Testing Safety-Critical SystemsSHARD Guide
Words
Flow Flow Provision Provision Failure Categorization Timing Failure Categorization Timing Value Value
Protocol Type Omission Commission Early Late Subtle Coarse
Pool Boolean No update Unwanted Update N/A Old data Stuck at N/A
Value No update Unwanted Update N/A Old data Wrong tolerance Out of tolerance
Complete No update Unwanted Update N/A Old data Incorrect Inconsistent
Channel Boolean No data Extra data Early Late Stuck at N/A
Value No data Extra data Early Late Wrong tolerance Out of tolerance
Complete No data Extra data Early Late Incorrect inconsistent
71
9.9 Testing Safety-Critical SystemsCleanroom
Control Structures and Correctness Conditions
  • Control structures Correctness conditions
  • Sequence For all arguments
  • f
  • DO
  • g Does g followed by h do f?
  • h
  • OD
  • Ifthenelse
  • f
  • IF p Whenever p is true
  • THEN does g do f, and
  • g whenever p is false
  • ELSE does h do f?
  • h
  • FI
  • Whiledo
  • f Is termination guaranteed, and
  • WHILE p whenever p is true
  • DO does g followed by f do f, and

72
9.9 Testing Safety-Critical SystemsA Program and
Its Subproofs
  • Program Subproofs
  • f1 f1 DO g1g2f2 OD ?
  • DO
  • g1
  • g2
  • f2 f2 WHILE p1 DO f3 OD ?
  • WHILE
  • p1
  • DO f3 f3 DO g3f4g8 OD?
  • g3
  • f4 f4 IF p2 THEN f5 ELSE f6 FI ?
  • IF
  • p2
  • THEN f5 f5 DO g4g5 OD ?
  • g4
  • g5
  • ELSE f6 f6 DO g6g7 OD ?
  • g6
  • g7

73
9.9 Testing Safety-Critical SystemsSidebar 9.12
When Statistical Usage Testing Can Mislead
  • Consider fault occurs for each
  • saturated condition 79 of the time
  • non saturated condition 20 of the time
  • transitional condition 1 of the time
  • probability of failures 0.001
  • To have a 50 chance of detecting each fault, we
    must run
  • non saturated 2500 test cases
  • transitional 500,000 test cases
  • saturated 663 test cases
  • Thus, testing according to the operational
    profile will detect the most fault
  • However, transition situation are often the most
    complex and failure-prone
  • Using operational profile would concentrate on
    testing the saturated mode, when in fact we
    should be concentrating on the transitional fault

74
9.10 Information Systems ExampleThe Piccadilly
System
  • Many variables, many different test cases to
    consider
  • An automated testing tool may be useful

75
9.10 Information Systems ExampleThings to
Consider in Selecting a Test Tool
  • Capability
  • Reliability
  • Capacity
  • Learnability
  • Operability
  • Performance
  • Compatibility
  • Nonintrusiveness

76
9.10 Information Systems ExampleSidebar 9.13 Why
Six-Sigma Efforts Do Not Apply to Software
  • A six-sigma quality constraint says that in a
    billion parts, we can expect only 3.4 to be
    outside the acceptable range
  • It is not apply to software because
  • People are variable, the software process
    inherently contains a large degree of
    uncontrollable variation
  • Software either conforms or it does not, there
    are no degree of conformance
  • Software is not the result of a mass-production
    process

77
9.11 Real-Time ExampleAriane-5 Failure
  • Simulation might help preventing the failure
  • Could have generated signals related to predicted
    flight parameters while turntable provided
    angular movement

78
9.12 What This Chapter Means for You
  • Should anticipate testing from the very beginning
    of the system life cycle
  • Should think about system functions during
    requirement analysis
  • Should use fault-tree analysis, failure modes and
    effect analysis during design
  • Should build safety case during design and code
    reviews
  • Should consider all possible test cases during
    testing
Write a Comment
User Comments (0)
About PowerShow.com