Searchbased Testing of Service Level Agreements

1
Search-based Testing ofService Level Agreements

• M. Penta, G. Canfora, G. Esposito, V. Mazza and
  M. Bruno
• U. of Sannio
• _at_ GECCO 07

2
Contents

• Research Issue and Objective
• Motivating Example
• The Proposed Algorithm
• QoS-risky paths
• Evolving test cases by leveraging GA
• Measurements
• Conclusion

3
Introduction

• In Service Oriented Architecture (SOA), an
  application is composed of services and a
  business process
• Each service encapsulates the function of an
  application component
• Each business process defines how services
  interact to achieve a certain business goal
• A business process refers abstract services, and
  they are bound to concrete services at runtime
• E.g., Trip Arrange Service is bound to expedia,
  cheaptickets,
• Different concrete service may have different QoS
  values (e.g., processing time, cost, )
• An application often offers its Service Level
  Agreement (SLA) to guarantee a certain QoS level
  (e.g., worst case processing time/throughput and
  longest downtime) to application users
• A SLA is included in a contract, and an
  application provider pays a penalty if its
  violated

4
Research Issue

• Before offering a SLA, an application provider
  need to limit the possibility that the SLA is
  violated
• ? Need to test an application whether it violates
  a SLA or not
• A SOA application may violate a SLA due to the
  combination of three factors
• Input data
• Input data may change the path in a business
  process
• Bindings between abstract and concrete services
• Network configuration and server load
• (not considered in this paper)
• Issue It takes long time to test the all
  combinations of factors
• E.g., a process has five abstract services and
  each has three concrete services ? 35 243
  combinations of concrete services
• How to reduce the time to test an application?

concrete services
Input data
A
abs. service
B
B
B
C
B
D
app
Output data
5
Research Objective

• Using Genetic Algorithms to generate (evolve)
  test cases causing SLA violations
• Each test case consists of input data and a
  combination of concrete services
• The proposed testing framework actually runs
  tests, evaluates the results, and generates the
  next test cases
• Without testing all combinations, the proposed
  algorithm generates test cases that violates a
  SLA in a short time

6
Assumption

• An abstract service is bound to one of concrete
  services at runtime
• A list of concrete services are given
• Concrete services for a certain abstract service
  provides the same function, but have different
  QoS values
• Each concrete service knows its QoS value
  estimates
• Estimates can be used to guess applications QoS
  values, but the measured values may be different.
  It need to run test to know the actual QoS values.

7
Motivating Example

• Image Processing App
• Scale, Posterize, Shapen and Gray abstract
  services

• Each abstract service has three concrete services
  with different QoS properties
• Resolution and response time
• Input data consists of
• To posterize? (posterize)
• Width and height (dim1 and dim2)
• Max are predefined
• of times to apply Sharpen (nsharpen)
• Max is predefined

• Dependeing on input data (path) and bindings of
  concrete services, applications QoS values may
  vary

8
Overview of the proposed Approach

• Determine a QoS-risky path
• Based on QoS value estimates of each service, the
  proposed framework identifies a path(s) that has
  highest chance to violate a SLA
• A QoS-risky path may violates a SLA already
• In that case, the algorithms stops ? developers
  revise the SLA
• Even if a QoS-risky path does not violate a SLA,
  it does not guarantee that the application never
  violate the SLA
• Because QoS value estimates are just estimates,
  and
• Paths similar to a QoS-risky path may validate a
  SLA
• Evolving Test Cases
• The proposed framework performs genetic
  operations to generate new set of test cases, and
  runs and evaluates them
• Directional evolution the framework tries to
  find paths similar to a QoS-risky path since a
  QoS-risky path has the highest chance to violate
  a SLA
• During an evolution, paths similar to a QoS-risky
  path are evaluated
• Repeat step 2 until a test case violating a SLA
  found

9
1) Determining a QoS-risky path

• Identify which workflow paths are likely to have
  high values for upper-bounded QoS properties
  (e.g., response time) and lower values for
  lower-bounded QoS properties (e.g., resolution)
• Identify all paths, and check all combinations of
  concrete services in each path
• E.g., Sum up all response times ? the worst
  response time of the path
• E.g., Pick the minimum resolution ? the worst
  resolution that the path supports
• This process does not run test, but uses QoS
  value estimates. So, it can be done in a short
  time. However, they are just estimations. So, it
  needs to run tests actually at last
• Normalize and sum up the differences between a
  SLA and estimates, and the smallest one is a
  QoS-risky path
• S(QoS property i in a SLA) (QoS property i of
  the path j)/QoS property i in a SLA) (the min
  of QoS property i among all paths)
• For upper-bounded QoS properties

10
2) Generating Test Cases

• The proposed framework performs genetic
  operations to generate new set of test cases
• A gene is composed of two data structures 1) a
  set of input data, and 2) a set of service
  bindings
• The initial genes are generated randomly

posterize
dim1
dim2
nsharpen
Input data
Scale
Posterize
Sharpen
Gray
Service bindings

• Input data may contain a list of data
• E.g, files variable contains a list of file URLs
  to process

11
Genetic Operations

• Mutation
• Bindings randomly change a binding
• Inputs integers and floats are replaced by
  random values in a predefined range. Strings are
  replaced by one of predefined strings. For a
  list, an element may be removed or inserted.
• Corssover
• Bindings one-point crossover at a random point
• Inputs swap randomly selected lists

12
Fitness Function

• Fitness function considers two objectives
• How far the test case j is from a SLA violation
  (Dj)
• S(QoS property i in a SLA) (measured property
  i of the path j)
• Assume measured properties do not violate
  constraints
• The smaller, the better
• Since the goal is to find a test case violating a
  SLA
• How much the test case j closes to a QoS-risky
  path (Pj)
• The smaller, the better
• Since paths similar to QoS-risky paths have
  higher chance to violate a SLA
• Pj dependent executedj Sbranch_distancej
• dependent the number of services in a QoS-risky
  path
• executedj the number of services executed by the
  test case j
• branch_distancej the distance from satisfying a
  condition of a branch to follow a QoS-risky path
• When a condition is satisfied, branch_distance
  0
• If a path follows a QoS-risky path,
  Sbranch_distancej 0
• When a condition is not satisfied, 0 lt
  branch_distance lt 1
• The larger, the farther

branch distance 0
branch distance gt 0
QoS-risky path
13
Branch Distance
Branch distance 0 if posterize true Branch
distance 1 if posterize false

• Branch distance is normalized in 0, 1
• The first condition posterize true
• Since the condition uses single boolean value,
  there is no intermediate state between two
  branches
• Selected ? distance 0
• Not selected ? distance 1
• The second condition dim1 dim2
• A branch distance depends on how dim1 and dim2
  close with each other
• Selected ? dim1 dim2 ? distance 0
• Not selected ? dim1 ! dim2 ? the closer, the
  smaller the distance is
• The third condition nsharpen gt 0
• Selected ? nsharpen gt 0 ? distance 0
• Not selected ? nsharpen lt 0 ? the smaller, the
  larger the distance is

Branch distance Norm(abs(dim1-dim2))
Branch distance Norm(abs(nsharpen)) if
nsharpen lt 0 Branch distance 0 if nsharpen
gt 0
14
Fitness Function
Fj current_gen/total_gen Dj (total_gen
current_gen)/total_gen Pj

• Fitness Function Fj changes weight values of Dj
  and Pj during evaluations
• total_gen number of max generations (predefined)
• current_gen current generation
• The fitness gives more weight to Pj, i.e., the
  distance from a QoS-risky path, at the first half
  of an evaluation. And gives more weight to Dj,
  i.e., the distance from a SLA violation, at the
  second half of an evaluation.
• First, the algorithm tries to find paths with
  high chance to violate a SLA. If it cannot find a
  path violates a SLA, then it tries to find
  combinations of service bindings to violates a
  SLA.

15
Evaluation

• An audio processing application, consists of
• Converter converts audio formats
• Encoder encode a PCM format in wav format
• Both are raw audio formats. PCM is for CD, wav is
  for PC.
• Decoder decodes a wav format in PCM
• Amplitude amplifies the audio signal
• Each abstract service has three concrete services
• QoS properties consist of processing time and bit
  rate, but details are not given
• Input Data
• converter boolean
• amplitudeTimes integer
• amplitude integer (specifies volume?)
• decoder string, D1 or D2 (D1 do decode, D2
  do not decode)
• encoder string, E1 or not (E1 do encode, else
  do not encode)
• QoS constraints
• Processing time (upper-bound) 12 seconds
• Bit rate (lower-bound) 65Kbps

16
QoS-risky path
17

• GA parameters
• Population (test cases) size 70 individuals
• Best 7 individuals are selected as parents
• Each creates 9 replicas??
• Max generations 100
• Max of evaluations (tests) 70 100
• Mutation probability 0.1
• Crossover probability 0.7
• The stopping criteria could just be a SLA
  violation, however the measurement in this paper
  keeps running
• To see how fitness values change

18

• The fitness here is 1/min(Fj)
• The proposed algorithm significantly outperforms
  a random search
• The paper does not say when a SLA violation is
  found

19

• White box considers QoS-risky paths (the
  proposed algorithm)
• Black box do not consider QoS-risky paths
• Only consider Dj, i.e., the distance from a SLA
  violation, to evaluate individuals
• E.g., a user cannot see the internal structure
  (services and a business process)
• There are several researches that do not consider
  the internal structure
• White box approach can quickly violate QoS
  constraints, and QoS values reach higher (in case
  of upper-bound properties) than in a black box
  approach

Response Time
Bit Rate
20
Conclusion

• The authors propose an algorithm/framework to
  test a SLA
• By leveraging GA to find test cases causing SLA
  violations
• (comments)
• Once a framework measures QoS properties of each
  services, it does not need to run actual tests
  anymore ??
• This paper assumes that QoS properties do not
  change at runtime
• Once all QoS properties are measured, it is
  possible to examine every combination off-line
• They should compare the proposed algorithm and a
  simple search such as 1) find a QoS-risky path,
  2) test it, 3) repeat this