Black-Box Testing

1
Black-Box Testing
( FÅP First-year Project Course, ITU, Denmark )

• Claus Brabrand
• brabrand_at_itu.dk

2
Reminder Learning Exam Goals

• Product
• Oral Exam

3
Outline

• Introduction
• Bugs and Black-box testing
• Testing the Specification
• static black-box testing
• Testing the Code
• dynamic black-box testing Equivalence
  partitioning
• Other techniques
• Model-checking Static-analysis
• Exercises
• Work on your group exercises

4
Definition bug

• Main entry 2bug
• Pronunciation /b?g/
• Function noun
• Etymology origin unknown
• Date 1622
• 1 a an insect or other creeping or crawling
  invertebrate (as a spider or centipede)
• b any of several insects (as the bedbug or
  cockroach) commonly considered obnoxious
• c any of an order (Hemiptera and especially
  its suborder Heteroptera) of insects that have
  sucking mouthparts, forewings thickened at the
  base, and incomplete metamorphosis and are often
  economic pests called also true bug
• 2 an unexpected defect, fault, flaw, or
  imperfection ltthe software was full of bugsgt
• 3 a a germ or microorganism especially when
  causing disease b an unspecified or nonspecific
  sickness usually presumed due to a bug
• 4 a sudden enthusiasm
• 5 enthusiast lta camera buggt
• 6 a prominent person
• 7 a crazy person
• 8 a concealed listening device
• 9 a weight allowance given apprentice jockeys

5
Example of a Bug -)
Hej Claus, A propos test og datoberegninger
Fredag morgen mente min PDA kalender (Microsoft
Windows CE) at vi skrev lørdag 1. marts. Jeg
stillede selvfølgelig tålmodigt uret tilbage til
fredag 29. februar 2008 (dette fandt sted på et
møde hos Microsoft i Vedbæk -) Lørdag formiddag
startede den så med at vise min (ret tomme)
kalender for august 2049. Det indbyggede ur stod
på 1. januar 1601. Ahem. Peter
6
Terminology (for Bugs)

• Generic terms
• Problem
• Error
• Bug (? well use this term)

• Severe conditions
• Fault
• Failure
• Defect

• Unintended operation
• Anomaly
• Incident
• Variance
• Feature (intended)

Typically imply blame as in- it was his fault
Famous quote - Its not a bug, its a feature
7
The Harvard Mark II Bug
The first documented computer bug was a moth
found trapped between points at Relay 70,
Panel F, of the Mark II Aiken Relay Calculator
while it was being tested Harvard University,
Sep. 9, 1947
Photo of firstactual bug
8
Definition Bug

• A bug is a relation between
• Specification
• Implementation
• Whether or not specification is
• Explicit or Implicit
• Written or Oral
• Formal or Informal
• (e.g., product spec vs. back-of-envellope)

9
Specification

• A spec states things an impl
• Should do!
• Shouldnt do!
• Unspecified? (unclear, unmentioned, or
  left open)

Should do!
Unspecified?
S
Shouldnt do!
Specs are often intentionally under-specified.
Its often better to not prematurely commit to
a particular solution (by specifying exactly how
a task should be done) and instead just state
which overall tasks the system should do.
10
Formal Definition Bug
cf. Software Testing, R.Patton, p.15

• A (software) bug occurs when
• 1. Impl doesnt do sth spec says it should
• 2. Impl does sth spec says it shouldnt
• 3. Impl does sth spec doesnt mention
• 4. Spec doesnt mention sth, but should
• 5. Impl is hard to understand/use by user(s)

(or does it incorrectly)
11
Consequences

• Consequently,
• Additional functionality?
• e.g., a calculator which also does square root
  (but wasnt supposed to)
• Easter egg?
• e.g., hit AltShift2 in Solitaire to win game

is a bug !
is a bug !
12
Question

• Question
• A) R.Patton interprets as (spec impl)
• ? trivially no bugs in impl !
• (according to own definition 1,2,3)
• B) More sensible to interpret as no spec
• ? entire impl is essentially one big bug !
• (according to definition 34 p.15)

an impl w/o a spec ?
Because the software is already complete, you
have the perfect specification the product
itself
R.Patton, Software Testing, p.31
13
Number one Cause of Bugs
cf. Software Testing, R.Patton, p.17

• is the specification

other
impl
spec
design

• Often due to
• complexity
• tight schedule
• under-specification
• under-documentation

(NB no percentages given)
14
Cost of (Fixing) Bugs
cf. Software Testing, R.Patton, p.18

• Cost of bugs increases exponentially (over time)

log
10,000,000
1,000,000
10,000
100
1
spec
design
code
test
release
15
Testers Goal
cf. Software Testing, R.Patton, p.19

• The goal of a software tester is
• Testers (therefore) often arent the most popular
  members of a project team

- 1) to find bugs - 2) find them as early as
possible and - 3) make sure they get
fixed R.Patton, Software Testing, p. 19
16
The Pesticide Paradox

• The pesticide paradox

• The more you test a software, the more immune
  it becomes to your tests
• B.Beizer, Software Testing Techniques, 1990

17
Bugs

• Bugs often occur in groups
• If you find one, chances are there are more
  nearby
• Not all bugs will be fixed
• Too costly?
• Too risky?
• Not cost-effective?
• Bug unknown -)

18
Programming vs. Testing

• Constructive thinking
• Test-to-pass
• Often not a good idea to test your own code -(

• Destructive thinking
• Test-to-fail
• (aka., error-forcing)
• Often better to test/break someone elses code -)

Advice test-to-pass before test-to-fail (i.e
., test normal usage first)
19
Black-Box Testing

• The goal of black-box testing is
• Make sure impl solves problem it is supposed to
• i.e.,
• Point of departure
• spec, not impl
• not a particular program which claims to solve
  problem
• testing w/o insights of code

impl spec
Static (test spec)
Dynamic (test impl)
20
Outline

• Introduction
• Bugs and Black-box testing
• Testing the Specification
• static black-box testing
• Testing the Code
• dynamic black-box testing Equivalence
  partitioning
• Other techniques
• Model-checking Static-analysis
• Exercises
• Work with your group exercises

21
Spec Warning Words (I/III)
cf. Software Testing, R.Patton, p.61

• Unconditionals (universally)
• Always, for every, for each, for all,
• Try to violate (i.e., find exemptions from rule)!
• Unconditionals (never)
• None, never, not under any circumstances,
• Try to violate (i.e., find exemptions from rule)!
• Tautologies
• Trivially, certainly, clearly, obviously,
  evidently,
• Check assumptions (that nothings swept under the
  rug)!

22
Spec Warning Words (II/III)
cf. Software Testing, R.Patton, p.61

• Unspecified conditionals
• Some(-times), often, usually,
  ordinarilly, mostly,
• Unclear spec (under which circumstances)?
• Continuations
• Etcetera, and so forth, and so on,
• Check that spec is comprehensively unambiguous?
• Examples
• E.g., for example, such as,
• Is example representative (what about other
  examples)?

23
Spec Warning Words (III/III)
cf. Software Testing, R.Patton, p.61

• Positive adjectives
• Good, fast, efficient, small, reliable,
  stable,
• Subjective (needs objectification if to be used
  for testing)!
• Alegedly completed
• Handled, processed, taken care of,
  eliminated,
• Is something hidden?
• Incompleted
• Skipped, unnecessary, superfluous,
  rejected,
• Is something forgotten?

24
Spec Warning Words (if-then)
cf. Software Testing, R.Patton, p.61

• Finally, watch out for
• If Then (with missing Else)
• Check what happens in the Else-case

• IF
• THEN
• ELSE ?!

25
Test Sequencing

• Question which is best (and why)?
• A) white-box testing black-box testing
  
• (i.e., white-box testing first)
• B) black-box testing white-box testing
  
• (i.e., black-box testing first)
• Answer B
• Settle overall impl spec problems first
• Before zooming in on the impl as in white-box
  testing

or
26
Outline

• Introduction
• Bugs and Black-box testing
• Testing the Specification
• static black-box testing
• Testing the Code
• dynamic black-box testing Equivalence
  partitioning
• Other techniques
• Model-checking Static-analysis
• Exercises
• Work on your group exercises

27
Test Boundaries

• Programs are vulnerable around the edges
• e.g. testing legal inputs (time, in hours)
• e.g. testing legal inputs (dates, in April)

Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Expected output invalid valid valid valid invalid
Actual output
Input -1 0 11 (e.g.) 23 24
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Input 00/4 01/4 12/4 (e.g.) 30/4 31/4
Expected output invalid valid valid valid invalid
Actual output
28
Test Powers-of-Two

• Programs vulnerable around powers-of-two
• e.g. years of age (assume held in a byte)
• e.g. game-spectators (assume held in a 16-bit
  word)

Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Expected output invalid valid valid valid invalid
Actual output
Input -1 0 27 (e.g.) 255 256
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Input -1 0 12345 (e.g.) 65535 65536
Expected output invalid valid valid valid invalid
Actual output
29
Test Empty Input

• Default / empty / blank / null / zero / none / ?
• e.g., any program

Property No input
Expected output Error message
Actual output
Input ?
30
Test Invalid Input

• Invalid / illegal / wrong / garbage / bogus data
• e.g., calculator

Property Invalid input Bogus data!!!
Expected output Error message Error message
Actual output
Input 31 ?!
31
Equivalence Partitioning

• Partitions and
• Equivalence Relations

32
Testing Infinite process

• Recall testing is an incomplete process
• (i.e., testing cant prove absence of bugs)
• There are infinitely many possible inputs
• (hence, testing will take an infinite amount of
  time)

33
Crash course on Relations

• Relations
• Equivalence Relations

34
Relations

• Example1 even relation
• Written as as a short-hand for
• and as as a short-hand for
• Example2 equals relation
• Written as as a short-hand for
• and as as a short-hand for
• Example3 dist-btwn relation
• Written as
• as short-hand for

_even ? Z
_even 4
4 ? _even
5 ? _even
? Z ? Z
(2,2) ?
2 2
2 ? 3
(2,3) ?
? ? CITY ? Z ? CITY
(Aarhus, 310, Copenhagen) ? ?
35
Equivalence Relation

• Let be a binary relation over set A
• ? A ? A
• is an equivalence relation iff
• Reflexive
• Symmetric
• Transitive

?x?A x x
?
?x,y?A x y ? y x
?
?x,y,z?A x y ? y z ? x z
36
Exercise Eq.Rel.

• Which relations are equivalence relations?
• and which are not (and why not)?
• a) The "less-than-or-equal-to" relation ' ? '
• (n,m) n,m ? Z, n ? m
• b) The almost-total-relation-on-integers,
  (relating all numbers except 42, but relating 42
  with 42)
• (n,m) n,m ? ( Z \ 42 ) ? (42,42)
  
• c) The "is-congruent-modulo-three" relation
• (n,m) n,m ? Z, (n 3) (m 3)
• d) The "have-talked-together" relation
• (p,q) n,m ? PEOPLE, p and q have talked
  together
• e) The "is-in-the-same-group-as" relation
• (p,q) n,m ? PEOPLE, p and q are in same
  FÅP group

37
Eq. Rel. ? Partition

• Equivalence relation
• (A,A), (B,B), (A,B), (B,A), (P,P), (X,X),
  (Y,Y), (Z,Z), (X,Y), (Y,X), (X,Z), (Z,X),
  (Y,Z), (Z,Y)
• E.g.
• A B, P P, X X, X Z
• A P, B Y, P Z

• Partition

P
A
X
B
Z
Y
Canonical representatives A B
A, B P P X
Y Z X, Y, Z
Capture the same information i.e., notion of
equivalence
38
Testing Infinite process

• Recall testing is an incomplete process
• (i.e., testing cant prove absence of bugs)
• There are infinitely many possible inputs
• (hence, testing will take an infinite amount of
  time)

39
Equivalence Partitioning

• Partition input
• Finitary partition
• If finite categories (aka. equivalence
  classes)
• Here 3 zero, pos, neg
• We can now test all equivalence classes
• Using representative elements from each category

zero
0
1 2 3
-3 -2 -1
neg
pos
40
Test Sum (contd)

• We can now test all equivalence classes
• Using representative input from each category
• Sum (testing all equivalence classes)

Property Pos, Pos Neg, Pos Zero,Pos Pos, Neg Neg,
Neg Zero,Neg Pos, Zero Neg, Zero Zero,Zero
Expected output 3 1 5 -1 -17 -10 11 -12 0
Actual output
Input (1,2) (-3,4) (0,5) (6,-7) (-8,-9) (0,-10) (1
1,0) (-12,0) (0,0)
41
Frequent Partitions for Testing

• Numbers
• Positive, Negative, Zero
• Zero, One, Two, Many (aka. Greenlandic Numbers)
• Lists
• Length-0, Length-1, Length-2, Length-3
• Ascending-elements, Descending-elements,
  AscDesc-elements

Advice - consider how problem might be solved -
partition into qualitatively different
categories such that (likely same category ?
same error)
42
Outline

• Introduction
• Bugs and Black-box testing
• Testing the Specification
• static black-box testing
• Testing the Code
• dynamic black-box testing Equivalence
  partitioning
• Other techniques
• Model-checking Static-analysis
• Exercises
• Work on your group exercises

43
Model-checking

• Modelling
• Verification
• Model-checking

44
Program vs. Model World
(Java)
(modeling lang)
Program world
Model world
abstraction
P
M
concretization
Abstract
Concrete
45
Methodology Model-based Design

• Design abstract model
• Decompose model
• Reason/Test/Verify model
• individual parts and whole
• Recompose insights
• make model safe
• Impl. concrete program

abstract
REAL PROBLEM
MODEL
reason
?
?
test
?
?
verify
concretize
SAFE MODEL
SAFE PROGRAM
46
Model-checking
(Java)
(modeling lang)
Program world
Model world
1. P ? ?
3. M ? ?
2. abstract
P
M
4. verify!!
5. M ? !
7. P ? !
6. concretize
Abstract
Concrete
If models are finite, we can have a computer
test all possibilities.
Never send a human to do a machines job --
A.Smith (99)
47
Static Analysis

• Static Analysis
• Undecidability

48
Undecidability

• Most interesting properties are undecidable
• e.g., can program P have a type error (when run)
• Compilers use approximations (computed via
  static analyses)

no program can decide this line automatically
(in all cases)
?
incorrect
correct
incorrect
correct
safe (over-)approximation
49
Undecidability (self-referentiality)

• Consider "The Book-of-all-Books"
• This book contains the titles of all books that
  do not have a self-reference (i.e. don't contain
  their title inside)
• Finitely many books i.e.
• We can sit down figure out whether to include
  or not...
• Q What about "The Book-of-all-Books"
• Should it be included or not?
• "Self-referential paradox" (many guises)
• e.g. "This sentence is false"

"The Bible"
"War and Peace"
"Programming Languages, An Interp.-Based
Approach"
...
The Book-of-all-Books
?
?
50
Termination Undecidable!

• Assume termination is decidable (in Java)
• i.e. ? some program, halts PROG ? BOOL
• Q Does P0 loop or terminate...? )
• Hence "Termination is undecidable"
• ...for Java, C, C, Pascal, ?-Calculus, Haskell,
  ...

bool halts(Program p) ...
-- P0.java --
bool halts(Program p) ... Program p0
read_program("P0.java") if (halts(p0)
halts) loop() else halt()
51
Example Type Checking

• Will this program have type error (when run)?
• Type error ? ltEXPgt evaluates to true

void f() var b if (ltEXPgt) b
42 else b true /
some code / if (b) ...
Undecidable (in all cases)
i.e., what ltEXPgt evaluates to (when run)
52
Example Type Checking (contd)

• Hence, languages use static requirements
• All variables must be declared
• And have constant type (throughout the program)

void f() bool b / some code /
if (ltEXPgt) b 42 else b
true / some more code /
Static compiler error Regardless of what
ltEXPgtevaluates to when run
53
Outline

• Introduction
• Bugs and Black-box testing
• Testing the Specification
• static black-box testing
• Testing the Code
• dynamic black-box testing Equivalence
  partitioning
• Other techniques
• Model-checking Static-analysis
• Exercises
• Work on your group exercises