Workshop on Empirical Methods for the Analysis of Algorithms

1
Workshop on Empirical Methods for the Analysis of
Algorithms
Evaluation of Parallel Metaheuristics by
Enrique Alba and Gabriel Luque
2
Introduction

• Parallelism is an approach that allows to reduce
  the execution time and also to improve the
  quality of the solutions.
• Clear metrics are necessary that allow to measure
  the performance of the parallel optimization
  procedure and to compare with other (parallel)
  approaches.
• Currently, there are several parallel metrics but
  their meaning and utilization in the
  metaheuristic community is not homogeneous.
• We are interested in revising, proposing, and
  applying parallel performance metrics and
  guidelines to ensure the correctness of our
  conclusions.

3
Parallel Performance Metrics (I)Speedup

• The most important parallel measure is the
  speedup that computes the ratio between
  sequential and parallel execution times.
• Speedup taxonomy
• Strong speedup.
• Weak speedup
• Speedup with solution stop
• Versus Panmixia.
• Orthodox.
• Speedup with predefined effort.

4
Parallel Performance Metrics (II)Other Measures

• Efficiency
• Incremental efficiency
• Generalized incremental efficiency
• Scaleup
• Serial fraction

5
Inadequate Utilization of Parallel Measures (I)

• Computational effort evaluation
• Eliminates effects of implementation, software
  and hardware.
• Misleading in the field of parallel methods
• Evaluation time is not constant.
• The goal of parallel methods is (only) not the
  reduction of the number of evaluations but the
  reduction of time.
• Comparing means/medians
• We cannot compare two averages or medians
  directly but we must compare the statistical
  distributions of the data.
• Statistical tests
• Normal data Student t-test or ANOVA.
• Otherwise non parametric test (e.g.,
  Kruskal-Wallis).

6
Inadequate Utilization of Parallel Measures (II)

• Comparing algorithms with different accuracy
• We are comparing different things.
• E.g. comparing methods solving different
  problems or different instances.
• Comparing parallel versions vs. canonical serial
  ones
• We are comparing different algorithms.
• E.g. comparing different methods (e.g., pGA vs.
  pSA).
• Using a predefined effort
• We are indirectly imposing the execution time.
• It is incorrect to use it to measure speedup.

7
Examples (I)Panmictic vs. Orthodox Speedup

• Panmictic speedup provides superlinear values.
• Orthodox speedup are worse (sublinear) than
  panmictic one but fair and realistic.
• Both cases show a same trend but in other
  experiments the trends could even be
  contradictory.

8
Examples (II)Speedup with Predefined Effort

• The termination condition is based on a
  predefined effort (a maximum number of
  evaluations).
• The calculation of speedup is not appropriate
• Accuracy of the solution found is different.
• We have fixed the execution time, and then we
  can calculate a theoretical speedup

• timem cm evalm teval (where cm ? 1/m).
• Predefined effort eval1 evalm
• sm c1 / cm.

9
Examples (III)Other Parallel Metrics

• Efficiency makes easy the analysis of the
  speedup.
• We observe a moderate loss of efficiency when we
  increase the number of processors.
• Since the serial fraction is almost constant,
  that loss of efficiency is due to the limited
  parallelism.

10
Conclusions

• In this work we have considered the issue of
  reporting parallel experimental research with
  parallel metaheuristics.
• We have observed that speedup is the most
  important metric in this field but several
  definition of it can be used.
• Speedup can be only applied when all the methods
  find solutions of similar quality and in these
  cases, the most appropriate definition is the
  orthodox one.
• The utilization of other metrics (efficiency and
  serial fraction) is an interesting complement to
  perform a complete and fair comparison.

11
Reykjavik, Iceland, September 2006
Málaga (SPAIN)
Questions?