Verification

1
Verification
2
Outline

• What are the goals of verification?
• What are the main approaches to verification?
• What kind of assurance do we get through testing?
• How can testing be done systematically?
• How can we remove defects (debugging)?
• What are the main approaches to software
  analysis?
• informal vs. formal

3
Need for verification

• Designers make mistakes even if they are skilled
  and follow sound principles
• Everything must be verified, every required
  quality, process and products
• even verification itself

4
Properties of verification

• May not be binary (OK, not OK)
• severity of defect is important
• some defects may be tolerated
• May be subjective or objective
• e.g., usability
• Even implicit qualities should be verified
• because requirements are often incomplete
• e.g., robustness

5
Approaches to verification

• Two approaches Experiment analysis
• Experiment with behavior of product
• sample behaviors via testing
• goal is to find "counterexamples"
• dynamic technique
• Analyze product to deduce its adequacy
• analytic study of properties
• static technique

6
Testing and lack of "continuity"

• Testing samples behaviors by examining "test
  cases"
• Impossible to estimate behavior of software from
  a finite set of test cases
• No continuity of behavior
• it can exhibit correct behavior in infinitely
  many cases, but may still be incorrect in some
  cases

7
Verification in engineering

• Example of bridge design
• One test assures infinite correct situations

8
procedure binary-search (key in element
table in elementTable found out Boolean)
is begin bottom table-first top
table-last while bottom lt top loop if
(bottom top) rem 2 ? 0 then middle
(bottom top - 1) / 2 else middle
(bottom top) / 2 end if if key table
(middle) then top middle else
bottom middle 1 end if end
loop found key table (top) end
binary-search
if we omit this the routine works if the else is
never hit! (i.e. if size of table is a power of
2)
9
Goals of testing

• To show the presence of bugs (Dijkstra, 1987)
• If tests do not detect failures, we cannot
  conclude that software is defect-free
• Still, we need to do testing
• driven by sound and systematic principles

10
Goals of testing (cont.)

• Should help isolate errors
• to facilitate debugging
• Should be repeatable
• repeating the same experiment, we should get the
  same results
• this may not be true because of the effect of
  execution environment on testing
• because of nondeterminism
• Should be accurate

11
Theoretical foundations of testing
12
Definitions (1)

• P (program),
• D (input Domain),
• R (output domain or Range)
• P D ? R (may be partial)
• Correctness is defined by OR ? D ? R
• P(d) is correct if ltd, P(d)gt ? OR
• P is correct if all P(d) are correct

13
Definitions (2)

• FAILURE
• P(d) is not correct
• may be undefined (error state or hung or crash)
• or may be the wrong result
• ERROR (DEFECT or BUG)
• anything that may cause a failure
• typing mistake
• programmer forgot to test x 0
• FAULT
• incorrect intermediate state entered by program
• A FAULT happens only if the program has ERROR
• FAILURE occurs if a FAULT happens during
  execution

14
Definitions (3)

• Test case t
• an element of D
• Test set T
• a finite subset of D
• Test is successful if P(t) is correct
• Test set is successful if P correct for all t in T

15
Definitions (4)

• Ideal test set T
• if P is incorrect, there is an element of T such
  that P(t) is incorrect
• if an ideal test set exists for any program, we
  could prove program correctness by testing

16
Test criterion

• A criterion C defines finite subsets of D
• (i.e., test sets)
• C ? 2D
• A test set T satisfies C if it is an element of C
• Example
• C ltx1, x2,..., xngt n ? 3 ? ? i, j, k, (
  xilt0 ? xj0 ? xkgt0)
• What is missing in this set
  definition?
• lt-5, 0, 22gt is a test set that satisfies C
• lt-10, 2, 8, 33, 0, -19gt also does
• lt1, 3, 99gt does not

17
Properties of criteria (1)

• C is consistent
• for any pairs T1, T2 satisfying C, T1 is
  successful iff T2 is successful
• so either of them provides the same information
• C is complete
• if P is incorrect, there is a test set T of C
  that is not successful
• C is complete and consistent
• identifies an ideal test set
• allows correctness to be proved!

NOT DISCUSSED IN CLASS
18
Properties of criteria (2)

• C1 is finer than C2
• for any program P
• for any T1 satisfying C1 there is a subset T2 of
  T1 which satisfies C2

NOT DISCUSSED IN CLASS
19
Properties of definitions

• None is effective, i.e., no algorithms exist to
  state if a program, test set, or criterion has
  that property
• In particular, there is no algorithm to derive a
  test set that would prove program correctness
• there is no constructive criterion that is
  consistent and complete

NOT DISCUSSED IN CLASS
20
Empirical testing principles

• Find strategy to select significant test cases
• Significant has high potential of uncovering
  presence of error

21
Complete-Coverage Principle

• Try to group elements of D into subdomains D1,
  D2, , Dn where any element of each Di is likely
  to have similar behavior
• D D1 ? D2 ? ? Dn
• Select one test as a representative of the
  subdomain
• If Dj ? Dk ? ? for all j, k (partition), any
  element can be chosen from each subdomain
• Otherwise choose representatives to minimize
  number of tests, yet fulfilling the principle

22
Complete-Coverage Principle
example of a partition
23
Testing in the small

• We test individual modules
• BLACK BOX (functional) testing
• partitioning criteria based on the modules
  specification
• tests what the program is supposed to do
• WHITE BOX (structural) testing
• partitioning criteria based on modules internal
  code
• tests what the program does

24
White box testing

• derives test cases from program code

25
Structural Coverage Testing

• (In)adequacy criteria
• If significant parts of program structure are not
  tested, testing is inadequate
• Control flow coverage criteria
• Statement coverage
• Edge coverage
• Condition coverage
• Path coverage

26
Statement-coverage criterion

• Select a test set T such that every elementary
  statement in P is executed at least once by some
  d in T
• Assignments I/Os Procedure calls
• If an input datum executes many statements then
  try to minimize the number of test cases still
  preserving the desired coverage

27
Example
read (x) read (y) if x gt 0 then write
("1") else write ("2") end if if y gt 0 then
write ("3") else write ("4") end if  
ltx 2, y 3gt, ltx - 13, y 51gt, ltx 97, y
-17gt, ltx - 1, y - 1gt covers all
statements ltx - 13, y 51gt, ltx 2, y -
3gt is minimal
28
Weakness of the statement coverage criterion
if x lt 0 then x -x end if z x
ltx-3 covers all statements it does not
exercise the case when x is positive and the
then branch is not entered
29
Edge-coverage criterion

• Select a test set T such that every edge (branch)
  of the control flow is exercised at least once by
  some d in T
• this requires formalizing the concept of the
  control graph, and how to construct it
• Edges represent statements
• Nodes at the beginning and end of an edge
  represent entry into the statement and exit

30
Control graph construction rules

31
Simplification
a sequence of edges can be collapsed into just
one edge WHY?
32
Exemple Euclid's algorithm
begin read (x) read (y) while x ? y loop
if x gt y then x x - y else y
y - x end if end loop GCD x end
read(x)
reqd(y)
x ! y
GCDx
x gt y
x lt y
end loop
end
x x- y
yy-x
end if
33
Weakness of edge-coverageexample search for an
element in a table
found false counter 1 while (not
found) and counter lt number_of_items
loop if table (counter) desired_element then
found true end if counter counter
1 end loop if found then write ("the desired
element is in the table") else write ("the
desired element is not in the table") end
if    
test cases (1) empty table, (2) table with 3
items, second of which is the item to look
for. Can not discover the error (lt should be
replaced by )
34
Weakness of Edge Coverage
  if x ? 0 then y 5 else z z - x
end if if z gt 1 then z z / x else z
0 end if
ltx 0, z 1gt, ltx 1, z 3gt causes the
execution of all edges
But fails to expose the risk of a division by
zero
35
Condition-coverage criterion

• Select a test set T such that every edge of Ps
  control flow is traversed and all possible values
  of the constituents of compound conditions are
  exercised at least once
• it is finer than edge coverage

36
Path-coverage criterion

• Select a test set T which traverses all paths
  from the initial to the final node of Ps control
  flow
• it is finer than previous kinds of coverage
• however, number of paths may be too large, or
  even infinite (see while loops)
• Therefore, additional constraints must be
  provided

37
The infeasibility problem

• Syntactically indicated behaviors (statements,
  edges, etc.) are often impossible
• unreachable code, infeasible edges, paths, etc.
• Adequacy criteria may be impossible to satisfy
• manual justification for omitting each impossible
  test case
• adequacy scores based on coverage
• example 95 statement coverage

38
Further problem

• What if the code omits the implementation of some
  part of the specification?
• White box test cases derived from the code will
  ignore that part of the specification!

39
Black box testing

• derives test cases from specifications

40
Specification of an example Sorted file of
Invoices
The program receives as input a record describing
an invoice. (A detailed description of the
format of the record has also been given). The
invoice must be inserted into a file of invoices
that is sorted by date. The invoice must be
inserted in the appropriate position If other
invoices exist in the file with the same date,
then the invoice should be inserted after the
last one. Also, some consistency checks must be
performed The program should verify whether the
customer is already in a corresponding file of
customers, whether the customers data in the
two files match, etc.
41
Consider these cases for testing the invoice
system
An invoice whose date is the current
date An invoice whose date is before the
current date (This might be even forbidden by
law) This case, in turn, can be split into the
two following subcases An invoice whose
date is the same as that some existing
invoice An invoice whose date does not exist
in any previously recorded invoice Several
incorrect invoices, checking different types of
inconsistencies
42
Systematic black-box techniques

• Testing driven by logic specifications
• Pre-/ Post-conditions (we cover only this)
• Syntax-driven testing
• Decision table based testing
• Cause-effect graph based testing

43
Logic specification for insertion of invoice
record in a file
for all x in Invoices, f in Invoice_Files sorted_
by_date(f) and not exist j, k (j ? k and f(j)
f(k) insert(x, f) sorted_by_date(f) and for
all k (old_f(k) z implies exists j (f(j) z))
and for all k (f(k) z and z ? x) implies
exists j (old_f(j) z) and exists j (f(j). date
x. date and f(j) ? x) implies j lt pos(x, f)
and result º x.customer belongs_to customer_file
and warning º (x belongs_to old_f or x.date lt
current_date or ....)
Pre-condition
Post-condition
44
Apply coverage criterion to post-condition
Rewrite in a more convenient way
TRUE implies sorted_by_date(f) and for all k
old_f(k) z implies exists j (f(j) z) and
for all k (f(k) z and z ? x) implies exists j
(old_f(j) z) and (x.customer belongs_to
customer_file) implies result and not
(x.customer belongs_to customer_file and ...)
implies not result and x belongs_to old_y
implies warning and x.date lt current_date implies
warning and ....
No less, No more
45
Syntax-driven testing (1)
NOT PART OF THE COURSE

• Consider testing an interpreter of the following
  language expressed in BNF

ltexpressiongt ltexpressiongt lttermgt
ltexpressiongt - lttermgt
lttermgt lttermgt
lttermgt ltfactorgt
lttermgt / ltfactorgt
ltfactorgt ltfactorgt
ident (
ltexpressiongt)
46
Syntax-driven testing (2)
NOT PART OF THE COURSE

• Apply complete coverage principle to all grammar
  rules
• Generate a test case for each rule of the grammar
• note, however that the test case might also cover
  other rules
• Note the specification is formal, and test
  generation can be automated

47
The oracle problem

• How to define the correctness of the output we
  obtain?
• Oracles are required at each stage of testing
• Automated test oracles are required for running
  large amounts of tests
• Oracles are difficult to design - no universal
  recipe
• If x gt 0 then S1 else S2 endif
• test against x 2 and x 0
• How do you know if the outcome of S1 and S2 are
  correct?

48
Testing in the large

• Module (Unit) testing
• testing a single module
• Integration testing
• integration of modules and subsystems
• System testing
• testing the entire system
• Acceptance testing
• performed by the customer

49
Module testing

• Experimental environment needed to create the
  environment in which the module should be tested
• stubs
• modules used by the module under test
• driver
• module activating the module under test

50
Module (Unit) Testing

• Driver
• Usually main program that accepts data and passes
  to the module to be tested and prints relevant
  results.
• Stub
• Simulates a subroutine module that is called by
  the module to be tested
• Test harness
• A collection of drivers and stubs
• An automatic test-result checking with
  anticipated-result will accelerate the testing
  process.

51
Type sequence(max_size NATURAL) IS record
size INTGER range 0 max_size
0 contents array (1 .. Max_size) of
INTEGER end record The Stub Looks Like
This Procedure sort (seq in out sequence)
is -- unsorted data input Begin write
(the sequence to be sorted is the following
) for I in 1 .. Seq.size loop
write (seq.contents (I)) ---
write unsorted data for user end loop
write (enter the result of sorting the
sequence) for I in 1 .. Seq.size
loop read (seq.contents (I) )
-- user provides sorted data end
loop --- a safer version of the stub could
verify the consistency of --- the
user-supplied data with respect to procedure
specification End sort
52
Testing a functional module
Sets the values
Provides the values
53
Integration testing

• Big-bang approach
• first test individual modules in isolation
• then test integrated system
• Incremental approach
• modules are progressively integrated and tested
• can proceed both top-down and bottom-up according
  to the USES relation

54
Integration testing and USES relation

• A driver is a program that simulates the use of
  the module
• being tested.
• Sets the values of the shared data as they would
  be set
• in the real application by other modules
  that are yet to
• be designed

55
Example
M1 USES M2 and M2 IS_COMPOSED_OF M2,1,
M2,2
CASE 1 Test M1, providing a stub for M2 and a
driver for M1 Then provide an implementation for
M2,1 and a stub for M2,2
CASE 2 Implement M2,2 and test it by using a
driver, Implement M2,1 and test the combination
of M2,1 and M2,2 (i.e., M2) by using a
driver Finally, implement M1 and test it with M2,
using a driver for M1
56
Analysis
57
Analysis vs. testing

• Testing characterizes a single execution
• Verification by experimentation
• Analysis characterizes a class of executions it
  is based on a model
• They have complementary advantages and
  disadvantages

• Analyzing a system means inspecting it to
  understand its
• properties and capabilities
• Example of testing a car

58
Informal analysis techniquesCode walkthroughs

• Based on playing the computer operations
• Recommended prescriptions
• Small number of people (three to five)
• Participants receive written documentation from
  the designer a few days before the meeting
• Predefined duration of meeting (a few hours)
• Focus on the discovery of errors, not on fixing
  them
• Participants designer, moderator, and a
  secretary
• Foster cooperation no evaluation of people
• Experience shows that most errors are discovered
  by the designer during the presentation, while
  trying to explain the design to other people.

59
Informal analysis techniquesCode inspection

• Organizational aspects similar to code
  walk-through
• A reading technique aiming at error discovery
• Based on checklists e.g.
• use of uninitialized variables
• jumps into loops
• Non-terminating loops
• array indexes out of bounds
• mismatch between actual / formal parameters
• Writing a procedure that modifies a formal
  parameter
• Calling the procedure with a constant value as
  the actual parameter

60
Java Code Inspection Checklistby Praktikum
Software Engineering
61
Java Code Inspection Checklistby Praktikum
Software Engineering
62
Java Code Inspection Checklistby Praktikum
Software Engineering
63
ExampleLow-level Design ofABM Modules
64
Design of database schema(i.e., data format for
mydatabase.txt)
65
Example ABM User-interface
Running the Simplified ABM System
66
ABM ExampleExecutable Files for Different
Machines

• You can run the executable file on a PC Windows
  system , on an Apple computer, or on a Sun
  Solaris system (see below) and compare the
  operations of your system with this reference
  system.
• Note you should put both the "executable file"
  and "mydatabase.txt" in one directory (for Apple
  and Solaris systems you must change the mode of
  the file to be "executable") and then run it.
• Go to Lab 9 in the course web page and click on
• PC Windows
• Apple Macintosh
• Sun Solaris

67
Black Box Testing of ABM (Lab 9)
68
Random Testing for OO classes

• Class Account open() setup() deposit()
  withdraw() balance() summarize()
  creditLimit() close()
• Constraints first open account, do operations,
  finally close account.
• Minimum case open gt setup gt deposit gt withdraw gt
  close
• General case
• Open gt setup gt deposit gt deposit withdraw
  balance summarize creditLimit n gt withdraw gt
  close
• Random Testing
• R1 open gt setup gt deposit gt deposit gt balance
  gtsummarize gt withdraw gt close
• R2 open gt setup gt deposit gt withdraw gt deposit gt
  balance gt creditLimit gt withdraw gt close
• Partition Testing
• P1 (Change State) open gt setup gt deposit gt
  deposit gt withdraw gtclose
• P2 (No C S) open gt setup gt deposit gt summarize gt
  creditLimit gt withdraw gt close

69
InterClass Test Case Design(class collaboration
testing)

• Random test cases
• For each client class, use the list of operations
  to generate a series of random test sequences.
  The operations will send messages to other server
  classes
• For each message that is generated, determine the
  collaborator class and the corresponding
  operation in the server object.
• For each operation in the server object (that has
  been invoked by the messages sent from the client
  object) , determine the messages that it
  transmits.
• For each of the messages, determine the next
  level of operations that are invoked and
  incorporate those into the test sequence.

70
Example of ATM (ABM) machine
71
Example Test Cases for ATMATM --gt Bank

• Sequences of messages between ATM and Bank
• verifyAcct gt verifyPIN gt verifyPolicy gt
  withdrawReq depositReq acctInfo n
• Random test cases
• R3 verifyAcct gt verifyPIN gt depositReq
• R4 verfiyAccBank validAccValidationInfo gt
  verifyPINBank gt validPinValidationInfo gt
  depositReqBank gt depositAccount

72
Tests Derived from Behavior Model

• State diagrams model the dynamic behavior of a
  class and can be used to derive a sequence of
  tests to test a class and its collaborators.
• The test should traverse all states.
• S1 open gt setupAccnt gt deposit (initial) gt
  withdraw (final) gt close
• S2 open gt setupAccnt gt deposit(initial) gt
  deposit gt balance gt credit gt withdraw(final) gt
  close
• S3 open gt setupAccnt gt deposit(initial) gt
  deposit gt withdraw gt accntInfo gt withdraw(final)
  gt close

73
Scenario-based Testing

• Tests the overall behavior of the system through
  complex interactions (task scenarios or
  use-cases) between user and system.
• Example use-cases for a text editor
• Fix the final draft
• Print the entire document
• Move around in the document, changing certain
  pages
• As each page is changed, it is printed
• Sometimes a series of pages are printed.
• Print a new copy
• Open the document
• Select print in the menu
• Check if youre printing a page range if so,
  click to print the entire document
• Click on the print button
• Close the document

74
Outline of the ABM System Presentation

• Overview a short introduction about software
  engineering and why it is important as an
  engineering discipline different phases of the
  software life cycle process and the way we
  exercised in this class.
• Discus the benefits of having a good requirement
  specification and present the SRS that you
  obtained from RFP.
• Discuss how you generated your high-level design
  (SDS using Component Diagram) from SRS and what
  are its features.
• Discuss how you refined SDS to low-level design
  (SDS using modules)
• Present your implementation in Java and discuss
  how good it is.
• Present a short demo of your running ABM system
• Conclusion your experiences in team working
  what you gained from this software engineering
  exercise the labs and how you think these
  exercises will be useful for your future career.
  Suggestions to improve the lab.