Testing

1
Testing
Coming Wednesday well hold a Q/A lecture for
Milestone 2

• Hans-Arno Jacobsen

2
Testing vs. debugging

• Debugging is what you do when you know that the
  program is broken
• Testing is a determined, systematic attempt to
  break a program that you think is working
• Testing can determine the presence of bugs, but
  not their absence (E. Dijkstra)

3
The first bug?Testing is about finding bugs!
1947
discovered a moth stuck in a relay and thereby
impeding operation
Source Wikipedia//Debugging. A photo of the
allegedly first real bug, which was debugged
in 1947
4
Testing for defects bugs

• All software has bugs
• All programmers, even the best introduce bugs
• Testing is to ensure the quality of software by
  finding defects

5
Attitude

• Be destructive and try to break the system!!!
• Yes, here this is allowed.
• Dont show that the software is correct. Show
  that the software is incorrect.
• Testing can prove the existence of bugs, but it
  cannot show their absence.
• If test does not find a fault, then it failed.

6
Prevalent attitudes

• Many companies developers
• View testing as irritating overhead
• Assign poor or novice programmers to testing
• Result Many developers look down on Q/A folks
• But, in the end
• Typically spend over 50 of development budget on
  testing
• Create more code for testing than for product
• Successful companies
• Treat testing seriously
• Assign good programmers to testing
• Test early in development process to save money
• Even practice test-driven development

7
Test as you write your code

• As you write the program verify simple
  properties.
• The earlier a problem is found the better.
• Some techniques
• Test code at its boundaries
• Test pre and post conditions
• Use assertions
• Program defensively
• Check error return codes

Following examples draw from
8
Anything wrong?

• int i
• char sMAX
• for (i0 (si getchar()) ! \n i lt
  MAX-1 i)
• s--i \0
• Reads characters until it finds a newline or
  fills a buffer

s0
s2
si
Input is ECE 297 is great fun!\n
9
Input An empty line \n
s0

• int i
• char sMAX
• for (i0 (si getchar()) ! \n i lt
  MAX-1 i)
• s--i \0
• For input of newline would set s-1 \0
• If you had checked for boundary conditions, you
  might have found this bug

Input is \n
10
Try again using standard C idioms
Anything wrong?

• for (i0 i lt MAX-1 i)
• if ( (si getchar()) \n )
• break
• si \0
• Repeat the boundary test (\n as first input)
• Now check for inputs with one, two characters
• Are there other boundary conditions?
• Try long lines

11
Try no line as input, i.e., EOF
End of File

• for (i0 i lt MAX-1 i)
• if ( (si getchar()) \n )
• break
• si \0
• for (i0 i lt MAX-1 i)
• if ( (si getchar()) \n II si EOF)
• break
• si \0

Another problem still remains!
12
Boundary condition checking

• At one boundary
• Empty line, no line, 1, 2, 3 characters
• Long line or no newline
• At other boundary
• Array nearly full, exactly full, over-full (with
  and without arrival of newline)
• What should happen if buffer fills before a
  newline arrives?
• Apparent gap in the specification (needs thought)
• Helps to find off-by-one errors

13
Anything wrong?

• double avg(double a, int n)
• int i
• double sum
• sum 0.0
• for (i0 i lt n i)
• sum ai
• return sum / n

14
Pre- and post-conditions

• double avg(double a, int n)
• int i double sum
• sum 0.0
• for (i0 i lt n i)
• sum ai
• return sum / n

Guaranteed wrong would be to ignore the problem!

• What should avg do for
• n 0 (empty array)
• Empty array makes sense
• Average of empty array makes no sense
• n lt 0
• Maybe return n lt 0 ? 0.0 sum/n

15
Test pre- and post-conditions

• Verify that expected or necessary properties hold
  
• before pre-conditions
• E.g., check input values are in range
• after post-conditions
• E.g., check output values are in range
• Division by zero would result in an interrupt and
  program would stop

16
Use assertions ltassert.hgt

• Lets you add pre- and post-condition tests
• Upon failure aborts the program
• Should be reserved for unexpected failures
  without possibility to recover
• Assertion failed
• n gt 0, file ,
• line 6 Abort(crash)
• Use at interfaces (caller / callee contracts)

above example assert(ngt0) for (i0 i lt n
i)
17
Program defensively I

• Program with suspicion
• Assume input will be incorrect
• Assume open files will be closed (or vice versa)
• Explicitly check requirements
• Exhaustively test all conditionals
• Default branch in switch
• Are all possibilities exhausted in the
  if-statement?

ECE 297
18
Program defensively II

• If (grade lt 0 grade gt 100) // cant happen
• letter ?
• else if (grade gt90)
• letter A
• else
• Make sure program protects itself against
  incorrect use of illegal data
• Null pointers, out of range subscripts, division
  by zero,

19
Something else

• include ltstdio.hgt
• include ltstring.hgt
• void doit(void)
• char buf8
• gets(buf)
• int main(void)
• doit()

20
Something else
Stack frame

• include ltstdio.hgt
• include ltstring.hgt
• void doit(void)
• char buf8
• gets(buf)
• int main(void)
• doit()

What happens?
gt./a.out gtAAAAAAAAAAAA gt
char buf
RET MAINDOIT
21
Systematic testing

• Test incrementally
• Test simple parts first, then integrate
• Know what output to expect
• Verify conservation properties
• Compare independent implementations
• Measure test coverage

22
Develop many test harnesses I

• Utility Does program work for reasonable input
• Robustness stress Test range of operating
  conditions
• What happens with invalid input
• Test for failure conditions, not just correct
  operation
• Reliability (with long-running tests)
• How often does the system fail
• How long does it take to fix and bring back up
• Performance
• Quantification of systems behavior
• Latency, throughput, response time, memory use,

23
Develop many test harnesses II

• Develop harness early for
• Every function
• Every module
• Every component
• Every class
• Every subsystem
• Entire program / system
• Test
• Against specification (black box testing)
• Every code path (white box testing)
• Size of testing code larger than code being
  tested. But can be used each time code changes.
  Testing is often automated by re-running some or
  all tests.

Unit testing!
24
Testing get Milestone 1
MyCourses

• As part of the test create the table
• create the supporting data structures
• like connection and record
• Define the tests
• int get(MyCourses, ECE297, r, c)
• Test oracle 95
• int get(MyCourses, ECE299, r, c)
• Record does not exist
• int get(MyCourse, ECE297, r, c)
• Table does not exist

Key Mark
ECE297 95
ECE344 92
ECE451 87
Test correctness, incorrectness, and failure
cases.
ECE 297
24
25
Testing principles I

• Testing is process of finding errors
• Test early and continuously in development cycle
  (dont go for big bang)
• It is impossible to completely test non-trivial
  modules
• Testing requires creativity and hard work

26
Testing principles II

• Use independent testers, but require developers
  to include (unit) test cases
• Automate testing (e.g., nightly builds,
  continuous builds, see tool CruiseControl)
• If testing does not find a fault, it failed!

Tool tip http//cruisecontrol.sourceforge.net/
27
Code walkthrough, code review, code inspection

• Approach
• Team of 3-6 with broad range of expertise
• Each independently reads through code looking for
  possible errors, possible problems, bad coding
  style
• Meeting to jointly discuss questionable items
• Advantages
• Tends to be effective
• Others get to know and understand code
• Goal Detect problems, not fix them
• Many variations of the above
• But implementation of this in practice greatly
  varies in organizations

28
Consumers as testers

• Regardless of how well tested, end-user will
  uncover bugs (and they are good at it!) which is
  why Beta versions of software exist.
• Have process to
• Record, track, document, and prioritize bugs
• Bug tracking tool Bugzilla
• Reproduce bug in the testing environment
• Fix bug
• Include tests that trigger bug in regression test
  suit
• Regression test Harness that tests for absence
  of every bug ever found.

Tool tip http//www.bugzilla.org/
29
Five steps to testing

1. Decide what how to test
2. Decide on test vectors (inputs to use)
3. Determine expected behavior Test Oracle
4. Develop test harnesses
5. Execute tests
6. Compare results to Test Oracle

Steps 5 and 6 can often be automated.
Tool tip Check is a unit testing framework for C
or JUnit for Java - http//www.junit.org/
30
Recommended development cycle

• Develop the feature, fix the bug,
• Add new test cases to test new or fixed
  functionality
• Test system with existing and new test cases
• Fix any problems and regressions
• Test again until all tests pass
• Commit sources including the new test cases to
  the repository

Your partner may have checked in new code! No
worries about commit, it would fail. BUT did you
test the system again before committing again?
ECE 297
30
31
Check

• Check is a unit testing framework for C
• Tests are run in a separate address space, why?
• Already setup for you in the skeleton code
• See /test subdirectory
• Already integrated in build (i.e., Makefiles)
• Some tests already provided as examples
• A Tech Talk is devoted to Check
• http//check.sourceforge.net/

32
Unit testing I

• Associated with the lightweight methodology
  called Extreme Programming (roots are earlier)
• Develop tests based on interface of unit
  (function, module, system etc.)
• Develop tests and code incrementally

33
Unit testing II

• Benefits
• Allows one to think about how the interface
  should be designed for usage in the early stage
• Encourages to think about corner cases early on
• Provides a documented level of correctness
• Intersperse writing unit tests with writing code
• " test a little, code a little
• Taken to the extreme in test-driven development
  first write the test, then the code

34
Storage server unit testing?

• What are possible candidates for unit testing in
  our storage server?

35
Summary on suggested tools

• Continuous builds or nightly builds
• Cruiscontrol
• Bug tracking
• Bugzilla
• Unit testing
• Check (in our case)
• Measure test coverage
• gcov
• Profile program
• gprof