TQS Teste e Qualidade de Software Software Testing and Quality Levels of Testing testing along the s

1
TQS - Teste e Qualidade de Software(Software
Testing and Quality)Levels of Testing(testing
along the software lifecycle)
João Pascoal Faria jpf_at_fe.up.pt
www.fe.up.pt/jpf
2
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

3
Types of testing
(reprinted)
Level of detail
system
integration
unit
Accessibility
security
white box
black box
robustness
performance
usability
reliability
functional behaviour (correctness)
Characteristics
4
Test levels and the extended V-model of software
development
(reprinted)
Execute acceptance tests
Specify Requirements
Execute system tests
System/acceptance test plan test cases
review/audit
Requirements review
Specify/Design Code
System/acceptance tests
Execute integration tests
Design
Integration test plan test cases review/audit
Design review
Specify/Design Code
Integration tests
Execute unit tests
Code
Unit test plan test cases review/audit
Code reviews
Specify/Design
Code Unit tests
(source I. Burnstein, pg.15)
(revisited)
5
Test levels or phases
(reprinted)

• Unit testing
• Testing of individual program units or components
  
• Usually the responsibility of the component
  developer (except sometimes for critical systems)
• Tests are based on experience, specifications and
  code
• A principal goal is to detect functional and
  structural defects in the unit
• Integration testing
• Testing of groups of components integrated to
  create a sub-system
• Usually the responsibility of an independent
  testing team (except sometimes in small projects)
• Tests are based on a system specification
  (technical specifications, designs)
• A principal goal is to detect defects that occur
  on the interfaces of units
• System testing
• Testing the system as a whole
• Usually the responsibility of an independent
  testing team
• Tests are usually based on a requirements
  document (functional requirements/specifications
  and quality requirements)
• A principal goal is to evaluate attributes such
  as usability, reliability and performance
  (assuming unit and integration testing have been
  performed)
• Acceptance testing
• Testing the system as a whole
• Usually the responsibility of the customer
• Tests are based on a requirements specification
  or a user manual

6
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

7
Unit testing object oriented systems (instead of
procedure oriented systems)

• Level
• methods usually not tested in isolation because
  of encapsulation and dimension (too small)
• classes
• Coverage criteria (for class testing)
• Testing all methods
• Setting and interrogating all object attributes
• Exercising the object in all possible states
  (e.g. with based on state machine model)
• Challenges
• Inheritance - more difficult to design test cases
  as the information to be tested is not localised
• Encapsulation - more difficult to check the
  objects state (requires query methods)
• Polymorphism - methods of the super-class may
  have to be retested for each new subclass

8
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

9
Integration testing

• Testing of groups of components integrated to
  create a sub-system
• Usually the responsibility of an independent
  testing team (except sometimes in small projects)
• Integration testing should be black-box testing
  with tests derived from the specification
• A principal goal is to detect defects that occur
  on the interfaces of units
• Main difficulty is localising errors
• Incremental integration testing (as opposed to
  big-bang integration testing) reduces this
  problem

10
Test drivers and stubs

• Auxiliary code developed to support testing
• Test drivers
• call the target code
• simulate calling units or a user
• where test procedures and test cases are coded
  (for automatic test case execution) or a user
  interface is created (for manual test case
  execution)
• Test stubs
• simulate called units
• simulate modules/units/systems called by the
  target code

11
Incremental integration testing
12
Approaches to integration testing

• Top-down testing
• Start with high-level system and integrate from
  the top-down replacing individual components by
  stubs where appropriate
• Bottom-up testing
• Integrate individual components in levels until
  the complete system is created
• In practice, most integration involves a
  combination of these strategies
• Appropriate for systems with a hierarchical
  control structure
• Usually the case in procedural-oriented systems
• Object-oriented systems may not have such a
  hierarchical control structure

13
Big-bang integration testing
Component under test
Test driver
Interface under test
Tested component
Test stub
Tested interface
14
Bottom-up integration testing
Test sequence
15
Top-down integration testing
Test sequence
16
Collaboration integration testing
(source Robert V. Binder)
Time
17
Advantages and disadvantages

• Architectural validation
• Top-down integration testing is better at
  discovering errors in the system architecture
• System demonstration
• Top-down integration testing allows a limited
  demonstration at an early stage in the
  development
• Test implementation
• Often easier with bottom-up integration testing
• Test observation
• Problems with both approaches. Extra code may be
  required to observe tests

18
Interfaces types

• Parameter interfaces
• Data passed from one procedure to another
• Shared memory interfaces
• Block of memory is shared between procedures
• Procedural interfaces
• Sub-system encapsulates a set of procedures to be
  called by other sub-systems
• Message passing interfaces
• Sub-systems request services from other
  sub-systems

19
Interface errors

• Interface misuse
• A calling component calls another component and
  makes an error in its use of its interface e.g.
  parameters in the wrong order
• Interface misunderstanding
• A calling component embeds assumptions about the
  behaviour of the called component which are
  incorrect
• Timing errors
• The called and the calling component operate at
  different speeds and out-of-date information is
  accessed

20
Interface testing guidelines

• Design tests so that parameters to a called
  procedure are at the extreme ends of their ranges
• Always test pointer parameters with null pointers
• Design tests which cause the component to fail
• Use stress testing in message passing systems
• In shared memory systems, vary the order in which
  components are activated

21
Object oriented integration testing

• OO systems may not have a hierarchical control
  structure so conventional top-down and bottom-up
  integration tests may have little meaning
• An alternative approach cluster or collaboration
  integration
• A cluster consists of classes that are related,
  for example, they may work together (collaborate)
  to support a required functionality (or a use
  case) for the complete system
• Cluster testing Identify clusters using
  knowledge of the operation of objects and the
  system features that are implemented by these
  clusters

22
Approaches to cluster testing

• Use-case or scenario testing
• Testing is based on a user interactions with the
  system
• Has the advantage that it tests system features
  as experienced by users
• Thread testing
• Tests the systems response to events as
  processing threads through the system
• See button example
• Object interaction testing
• Tests sequences of object interactions that stop
  when an object operation does not call on
  services from another object

23
Scenario-based testing

• Identify scenarios from use-cases and supplement
  these with interaction diagrams that show the
  objects (including objects internal to the
  system) involved in the scenario
• Example consider the scenario in a weather
  station system where a report is generated

24
Collect weather data
25
Weather station testing

• Thread of methods executed
• CommsControllerrequest WeatherStationreport
  WeatherDatasummarise
• Inputs and outputs
• Input of report request with associated
  acknowledge and a final output of a report
• Can be tested by creating raw weather data and
  ensuring that it is summarised properly

26
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

27
System testing

• Testing the system as a whole
• Usually the responsibility of an independent
  testing team
• Often requires many resources, special laboratory
  equipment and long test times
• Usually based on a requirements document,
  specifying both functional and non-functional
  (quality) requirements
• Preparation should begin at the requirements
  phase with the development of a master test plan
  and requirements-based tests (black-box tests)
• The goal is to ensure that the system performs
  according to its requirements, by evaluating both
  functional behavior and quality requirements such
  as reliability, usability, performance and
  security
• This phase of testing is especially useful for
  detecting external hardware and software
  interface defects, for example, those causing
  race conditions, deadlocks, problems with
  interrupts and exception handling, and
  ineffective memory usage
• Tests implemented on the parts and subsystems may
  be reused/repeated, and additional tests for the
  system as a whole may be designed

28
Types of system testing
Fully integrated software system
Usage profile
System tests
Usability and accessibility tests
Security tests
Configuration tests
Functional tests
Stress and load tests
Performance tests
Reliability and availability tests
Recovery tests
Test team
...
System ready for acceptance test
tests applicable depend on the characteristics of
the system and the available test resources
29
Types of requirements

• Functional requirements
• Describe what functions the software should
  perform
• Compliance to these requirements tested at the
  system level with functional systems tests
• Quality requirements
• Nonfunctional in nature (also called
  non-functional requirements)
• Describe quality levels expected for the software
• The users and other stakeholders may have
  objectives for the system in terms of memory use,
  response time, throughput, etc.
• Should be quantified as much as possible
• Compliance to these requirements tested at the
  system level with nonfunctional systems tests

30
Functional testing

• Ensure that the behavior of the system adheres to
  the requirements specification
• Black-box in nature
• Equivalence class partitioning, boundary-value
  analysis and state-based testing are valuable
  techniques
• Document and track test coverage with a (tests to
  requirements) traceability matrix
• A defined and documented form should be used for
  recording test results from functional and other
  system tests
• Failures should be reported in test incident
  reports
• Useful for developers (together with test logs)
• Useful for managers for progress tracking and
  quality assurance purposes

31
Performance testing

• Goals
• See if the software meets the performance
  requirements
• See whether there any hardware or software
  factors that impact on the system's performance
• Provide valuable information to tune the system
• Predict the system's future performance levels
• Results of performance test should be quantified,
  and the corresponding environmental conditions
  should be recorded
• Resources usually needed
• a source of transactions to drive the
  experiments, typically a load generator
• an experimental test bed that includes hardware
  and software the system under test interacts with
• instrumentation of probes that help to collect
  the performance data (event logging, counting,
  sampling, memory allocation counters, etc.)
• a set of tools to collect, store, process and
  interpret data from probes

32
Stress and load testing

• Stress testing testing a system with a load
  that causes it to allocate its resources in
  maximum amounts
• Other definitions (Ron Patton)
• load testing maximize the load imposed on the
  system (number of concurrent users, volume of
  data, ...)
• stress testing minimize resources available to
  the system (processor speed, available memory
  space, available disk space, ...)
• usually one is interested in a combination of
  both
• The goal is to try to break the system, find the
  circumstances under which it will crash, and
  provide confidence that the system will continue
  to operate correctly (possibly with bad
  performance but with correct functional behavior)
  under conditions of stress
• Example a system is required to handle 10
  interrupts / second and the load causes 20
  interrupts/second
• Another example a suitcase being tested for
  strength and endurance is stomped by a multiton
  elephant
• Stress testing often uncovers race conditions,
  deadlocks, depletion of resources in unusual or
  unplanned patterns, and upsets in normal
  operation that are not revealed under normal
  testing conditions
• Supported by many of the resources used for
  performance testing

33
Configuration testing

• Typical software systems interact with multiple
  hardware devices such as disc drives, tape
  drives, and printers
• Objectives according to Beizer
• show that all the configuration changing commands
  and menus work properly
• show that all interchangeable devices are really
  interchangeable, and that they each enter the
  proper states for the specified conditions
• show that the systems' performance level is
  maintained when the devices are interchanged, or
  when they fail
• Types of test to be performed
• rotate and permutate the positions of devices to
  ensure physical /logical device permutations work
  for each device
• induce malfunctions in each device, to see if the
  system properly handles the malfunction
• indice multiple device malfunctions to see how
  the system reacts

34
Security testing

• Evaluates system characteristics that relate to
  the availability, integrity and confidentiality
  of system data and services
• Computer software and data can be compromised by
• criminals intent on doing damage, stealing data
  and information, causing denial of service,
  invading privacy
• errors on the part of honest developers/maintainer
  s (and users?) who modify, destroy, or compromise
  data because of misinformation,
  misunderstandings, and/or lack of knowledge
• Both can be perpetuated by those inside and
  outside on an organization
• Areas to focus password checking, legal and
  illegal entry with passwords, password
  expiration, encryption, browsing, trap doors,
  viruses, ...
• Usually the responsibility of a security
  specialist
• See Segurança em Sistemas Informáticos

35
Recovery testing

• Subject a system to losses of resources in order
  to determine if it can recover properly from
  these losses
• Especially important for transaction systems
• Example loss of a device during a transaction
• Tests would determine if the system could return
  to a well-known state, and that no transactions
  have been compromised
• Systems with automated recovery are designed for
  this purpose
• Areas to focus Beizer
• Restart the ability of the system to restart
  properly on the last checkpoint after a loss of a
  device
• Switchover the ability of the system to switch
  to a new processor, as a result of a command or a
  detection of a faulty processor by a monitor

36
Reliability and availability testing

• Software reliability is the probability that a
  software system will operate without failure
  under given conditions for a given interval
• May be measured by the mean time between failures
  (MTBF)
• MTBF MTTF (mean time to failure) MTTR (mean
  time to repair)
• Software availability is the probability that a
  software system will be available for use
• May be measured by the percentage of time the
  system is on or uptime (example 99,9)
• A MTTR / MTBF
• Low reliability is compatible with high
  availability in case of low MTTR
• Requires statistical testing based on usage
  characteristics/profile
• During testing, the system is loaded according to
  the usage profile
• More information Ilene Burnstein, section 12.5
• Usually evaluated only by high maturity
  organizations

37
Usability and accessibility testing

• See Interacção Pessoa Computador

38
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

39
Acceptance, alpha and beta testing

• For tailor made software
• acceptance tests performed by users/customers
• much in common with system test
• For packaged software (market made software)
• alpha testing on the developers site
• beta testing on a user site
• For more information Ilene Berstein, section 6.15

40
Index

• Introduction
• Unit testing
• Integration testing
• System testing
• Acceptance, alpha and beta testing
• Regression testing

41
Regression testing

• Not really a new level of testing
• Just the repetition of tests (at any level) after
  a software modification, to ensure that
  previously detected bugs have been corrected and
  new bugs have not been introduced
• Requires test automation, good configuration
  management and problem/bug tracking
• Safe attitude repeat all tests, and not only the
  ones that failed in the previous test cycle
• because the correction of a bug may produce new
  bugs!
• Problem how to decide safely which tests need
  not be repeated (to accelerate regression
  testing)?

42
References and further reading

• Practical Software Testing, Ilene Burnstein,
  Springer-Verlag, 2003
• Software Testing, Ron Patton, SAMS, 2001
• Software Engineering, Ian Sommerville,
  6th Edition, Addison-Wesley, 2000