Test this group using stubs to take the place of ..

1
Integration and Higher Level Testing
Software Testing and Verification Lecture 11

• Prepared by
• Stephen M. Thebaut, Ph.D.
• University of Florida

2
Context

• Higher-level testing begins with the integration
  of (already unit-tested) modules to form
  higher-level program entities (e.g., components).
• The primary objective of integration testing is
  to discover interface errors among the elements
  being integrated. (Somewhat in keeping with the
  spirit of smoke testing.)

3
Context (contd)

• Once the elements have been successfully
  integrated (i.e., once they are able to function
  together), the functional and non-functional
  characteristics of the higher-level element can
  be tested thoroughly (via component, product, or
  system testing).

4
Integration Testing

• Integration testing is carried out when
  integrating (i.e., combining)
• Units or modules to form a component
• Components to form a product
• Products to form a system
• The strategy employed can significantly affect
  the time and effort required to yield a working,
  higher-level element.

5
Integration Testing (contd)

• Note that integration testing is sometimes
  defined as the level of testing between unit and
  system. We use a more general model of the
  testing process.

6
Levels of Testing
A More General View
System Test
A Popular View
Integration Test
System Test
Product Test
Integration Test
Integration Test
Unit Test
Component Test
Integration Test
Unit Test
7
Integration Testing Strategies

• The first (and usually the easiest...) issue to
  address is the choice between instantaneous and
  incremental integration testing.
• The former is sometimes referred to as the big
  bang approach. (Guess why!) Locating subtle
  errors can be very difficult after the bang.
• Integration is a process, not an event.

8
Integration Testing Strategies (contd)

• Incremental integration testing results in some
  additional overhead, but can significantly reduce
  error localization and correction time. (What is
  the overhead?)
• The optimum incremental approach is inherently
  dependent on the individual project and the pros
  and cons of the various alternatives.

9
As an aside

• Heres a slide from an on-line lecture that I
  came
• across recently

Principles of Integration

• Proper policy and plans
• Advocacy
• Manpower training
• Realistic tasks

• Coordination with other
• sectors
• Proper support
• Access to drugs

Integration of Substance Abuse Disorders in
National Rural Health Mission (NRHM), Centre
for Community Medicine, All India Institute
of Medical Sciences, New Delhi
10
Incremental Strategies

• Suppose we are integrating a group of modules to
  form a component, the control structure of which
  will form a calling hierarchy as shown.

11
Incremental Strategies (contd)

• In what order should the modules be integrated?
• From the top (root) module toward the bottom?
• From bottom (leaf) modules toward the top?
• By function?
• Critical or high-risk modules first?
• By availability?

12
Incremental Strategies (contd)

• How many should be combined at a time?
• What scaffolding (i.e., drivers and stubs to
  exercise the modules and oracles to
  interpret/inspect test results) will be required?
• Is scaffolding ever required outside the context
  of integration testing?

13
Top-Down Strategy

• Start with the root and one or more called
  modules.
• Test this group using stubs to take the place of
  missing called modules, and one driver (if
  necessary) to call the root module.
• Add one or more other called modules, replacing
  and providing new stubs as necessary.
• (contd)

14
Top-Down Strategy (contd)

• Continue the process until all elements have been
  integrated and tested.

15
Top-Down Strategy (contd)
A
B
C
D
C
D
E
F
G
H
I
G
H
I
J
K
L
16
Top-Down Strategy (contd)
driver
A
B
stub
stub
stub
stub
17
Top-Down Strategy (contd)
driver
A
B
stub
C
C
stub
stub
stub
stub
18
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
stub
stub
stub
stub
stub
stub
19
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
stub
stub
stub
stub
stub
20
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
stub
stub
stub
stub
stub
21
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
stub
stub
stub
stub
22
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
stub
stub
stub
stub
23
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
I
I
stub
stub
stub
24
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
I
I
J
stub
stub
25
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
I
I
J
K
stub
26
Top-Down Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
I
I
J
K
L
27
Top-Down Strategy (contd)

• Potential Advantages
• Allows early verification of high-level behavior.
• One driver (at most) is required.
• Modules can be added one at a time with each step
  if desired.
• Supports both breadth first and depth
  first approaches.

28
Top-Down Strategy (contd)

• Potential Disadvantages
• Delays verification of low-level behavior.
• Stubs are required for missing elements.
• Test case inputs may be difficult to formulate.
• Test case outputs may be difficult to interpret.
  (Oracles may be needed to interpret/inspect test
  results.)

29
Bottom-Up Strategy

• Start at the bottom of the hierarchy with two or
  more sibling leaf modules, or an only-child leaf
  module with its parent.
• Test this group using a driver. (No stubs are
  required.)
• Add one or more additional siblings, replacing
  drivers with modules only when all modules they
  call have been integrated.
• (contd)

30
Bottom-Up Strategy (contd)

• Continue the process until all elements of the
  subtree have been integrated and tested.
• Repeat the steps above for the remaining subtrees
  in the hierarchy (or handle in parallel).
• Incrementally combine the sub-trees and then add
  the root module.

31
Bottom-Up Strategy (contd)
A
B
C
D
C
D
E
F
G
H
I
G
H
I
J
K
L
32
Bottom-Up Strategy (contd)
driver
F
J
33
Bottom-Up Strategy (contd)
driver
F
E
J
34
Bottom-Up Strategy (contd)
driver
B
F
E
J
35
Bottom-Up Strategy (contd)
driver
B
F
E
driver
J
K
L
36
Bottom-Up Strategy (contd)
driver
B
driver
F
E
H
H
J
K
L
37
Bottom-Up Strategy (contd)
driver
B
driver
F
E
H
H
I
I
J
K
L
38
Bottom-Up Strategy (contd)
driver
driver
B
D
D
F
E
H
H
I
I
J
K
L
39
Bottom-Up Strategy (contd)
driver
driver
B
D
D
driver
F
E
H
H
I
I
G
G
J
K
L
40
Bottom-Up Strategy (contd)
driver
driver
driver
B
D
D
C
C
F
E
H
H
I
I
G
G
J
K
L
41
Bottom-Up Strategy (contd)
driver
driver
D
D
C
C
B
H
H
I
I
G
G
E
F
K
L
J
42
Bottom-Up Strategy (contd)
driver
driver
B
C
C
D
D
F
E
G
G
H
H
I
I
J
K
K
L
43
Bottom-Up Strategy (contd)
driver
B
C
C
D
D
E
F
G
G
H
H
I
I
J
K
L
44
Bottom-Up Strategy (contd)
driver
B
C
C
D
D
E
F
G
G
H
H
I
I
J
K
L
45
Bottom-Up Strategy (contd)
driver
A
B
C
C
D
D
E
F
G
G
H
H
I
I
J
K
L
46
Bottom-Up Strategy (contd)

• Potential Advantages
• Allows early verification of low-level behavior.
• No stubs are required.
• Easier to formulate input data for some subtrees.
  
• Easier to interpret output data for others.

47
Bottom-Up Strategy (contd)

• Potential Disadvantages
• Delays verification of high-level behavior.
• Drivers are required for missing elements.
• As subtrees are combined, a large number of
  elements may be integrated at one time.

48
Hybrid Incremental Integration Approaches

• Risk Driven
• Start by integrating the most critical or
  complex modules together with modules they call
  or are called by.
• Schedule Driven
• To the extent possible, integrate modules as
  they become available.
• (contd)

49
Hybrid Incremental Integration Approaches (contd)

• Function or Thread Driven
• Integrate the modules associated with a key
  function (thread) continue the process by
  selecting another function, etc.

50
How about Object-Oriented Systems?

• Just as a calling hierarchy allows design of an
  integration strategy for imperative software,
  use/include relations serve this purpose for
  object-oriented software.
• Since there may be no single root class,
  testing usually proceeds cluster by cluster in a
  bottom-up fashion, starting with leaf classes
  that depend on no others.
• We will come back to this in Lecture 12.

51
Higher-Level Testing

• Higher-level tests focus on the core
  functionality specified for higher level
  elements, and on certain emergent properties that
  become more observable as testing progresses
  toward the system level.
• The black-box testing strategies already
  considered (e.g., partition and combinatorial
  approaches) apply to functional testing at any
  level.

52
Higher-Level Testing (contd)

• Higher-level testing typically includes
• A brief overview of each follows.

• Usability test
• Installability test
• Serviceability test
• Performance test
• Stress test
• Security test

• Software compatibility test
• Device and configuration test
• Recovery test
• Reliability test

53
Usability Test

• Focus is on factors which influence the ease with
  which potential end-users are able to utilize the
  system to accomplish their goals.
• Specialized and sophisticated HCI experts
  conduct experiments in simulated work
  environments.
• Protocol analysis is utilized to identify
  usability bottlenecks.
• Application-specific metrics related to
  under-standability, learnability, and operability
  may be employed.

54
Installability Test

• Focus is functional and non-functional
  requirements related to the installation of the
  product/system.
• Coverage includes
• Media correctness and fidelity
• Relevant documentation (including examples)
• Installation processes and supporting system
  functions.
• (contd)

55
Installability Test (contd)

• Functions, procedures, documentation, etc.,
  associated with product/system decommissioning
  must also be tested.

56
Serviceability Test

• Focus is requirements related to upgrading or
  fixing problems after installation.
• Coverage includes
• Change procedures (adaptive, perfective, and
  corrective service scenarios)
• Supporting documentation
• System diagnostic tools

57
Performance Test

• Focus is requirements related to throughput,
  response time, memory utilization, input/ output
  rates, etc.
• Also very specialized in some organizations
  sophisticated test-beds and instrumentation are
  the norm.
• Statistical testing based on an operational
  profile is often employed.
• Requirements must be stated in quantifiable terms.

58
Stress Test

• Focus is system behavior at or near overload
  conditions (i.e., pushing the system to
  failure).
• Often undertaken with performance testing.
• In general, products are required to exhibit
  graceful failures and non-abrupt perfor-mance
  degradation.

59
Security Test

• Focus is vulnerability of resources to
  unauthorized access or manipulation.
• Issues include
• Physical security of computer resources, media,
  etc.,
• Login and password procedures/ policies,
• Levels of authorization for data or procedural
  access, etc.

60
Software Compatibility Test

• Focus is compatibility with other
  products/systems in the environment and/or with
  interoperability standards.
• May also concern source- or object-code
  compatibility with different operating
  environment versions.
• AKA compatibility/conversion testing when
  conversion procedures or processes are involved.

61
Device and Configuration Test

• Focus is configurability for, and/or
  compatibility with, all supported hardware
  configurations.
• Particularly taxing for client/server-based
  applications
• Tests are usually limited to combinations of
  representative devices for each supported
  protocol.

62
Recovery Test

• Focus is ability to recover from exceptional
  conditions associated with hardware, software, or
  people.
• This can involve
• detecting exceptional conditions,
• switch-overs to standby systems,
• recovery of data,
• maintaining audit trails, etc.
• May also involve external procedures such as
  storing backup tapes, etc.

63
Reliability Test

• Requirements may be expressed as
• the probability of no failure in a specified time
  interval, or as
• the expected mean time to failure.
• Appropriate interpretations for failure and time
  are critical.
• Utilizes Statistical testing based on an
  operational profile.

64
Integration and Higher Level Testing
Software Testing and Verification Lecture 11

• Prepared by
• Stephen M. Thebaut, Ph.D.
• University of Florida