Taxonomy of Hybrid Metaheuristics

1
Taxonomy of Hybrid
Metaheuristics

• Presented by Xiaojun Bao
  Lijun Wang
• School of
  Engineering
• University of
  Guelph
• Paper review for ENGG6140 Optimization

2
Outline

• Introduction
• Design issues of Hybrid Metaheuristics
• Implementation issues of Hybrid Metaheuristics
• A Grammar for extended hybrid schemes
• Conclusion
• 6. Reference

3
Introduction

• 1.1 Single-solution algorithms
• - descent local search
• - greedy heuristic
• - simulated annealing
• - tabu search

4
Introduction

• 1.2 Metaheuristic algorithms
• - evolutionary algorithms

EA genetic algorithms
evolution strategies
genetic programming
5
Introduction

• - ant colonies
• - scater search
• - and so on.

6
Hybridization

• 1.3 Hybrid Metaheuristics
• So far, many hybrid metaheuristic algorithms have
  been proposed and implemented to solve many
  combinatorial optimization problems, e.g. those
  known as NP-hard.
• The best results found for various practical
  problems have proven that combination of
  different algorithms are very powerful, in case
  of large and difficult problems.

7
Hybridization

• A taxonomy of hybrid algorithms is presented to
  provide a common terminology and classification
  mechanism
• Based on the general taxonomy, we could make the
  comparison of the hybrid algorithms in a
  qualitative way.

8
Hybridization

• Hybridization of heuristics involves a few major
  issues which may be classified as design and
  implementation respectively.

functionality
design
architecture
9
Hybridization
hardware platform
programming model
implementation
environment
10
Design issues

• 2.1 Hierarchical classification
• The structure of the hierarchical portion of
  the taxonomy is shown as follows

11
Design issues
Figure 1. Classification (design issues)
12
Hierarchical classification

• 2.1.1 Low-level versus High-level
• The low-level hybridization addresses the
  functional composition of a single optimization
  method. In this hybrid class, a given function of
  a metaheuristic is replaced by another
  metaheuristic.

13
Hierarchical classification

• In high-level hybrid algorithms, the different
  metaheuristics are self-contained. We have no
  direct relationship to the internal workings of a
  metaheuristic.

14
Hierarchical classification

• 2.1.2 Relay versus Co-evolutionary
• In relay hybridization, a set of
  meta-heuristics is applied one after another,
  each using the output of the previous as its
  input, acting in a pipeline fashion.

15
Hierarchical classification

• Co-evolutionary hybridization represents
  cooperative optimization models, in which we have
  many parallel cooperating agents, each agent
  carries out a search in a solution space.

16
Hierarchical classification

• Four classes are divided from this
  hierarchical taxonomy
• LRH (Low-level Relay Hybrid).
• This class of hybrids represents
  algorithms in which a given metaheuristic is
  embedded into a single-solution metaheuristic.
  Few examples from the literature belong to this
  class.
• Let us look at the following example

17
Hierarchical classification

• Figure 2. An example of LRH hybridization
  embedding local search in simulated
• annealing

18
Hierarchical classification

• LCH (Low-level Co-evolutionary Hybrid)
• Two competing goals govern the design of a
  metaheuristic exploration and exploitation.
• In order to achieve the best performance,
  most efficient population-based heuristics (i.e.,
  genetic algorithms, scatter search, ant colonies,
  etc.) have been coupled with local search method
  such as hill-climbing, simulated annealing and
  tabu search.

19
Hierarchical classification

• An example
• Figure 3. LCH. For instance, a tabu search is
  used as a mutation operator and a greedy
  heuristic as a crossover operator in a genetic
  algorithm

GA Individuals individual Crossover
mutation
Greedy heuristic
tabu
20
Hierarchical classification

• HRH (High-level Relay Hybrid).
• In HRH hybrid, self-contained
  metaheuristics are executed in a sequence.
• For example, evolutionary algorithms are
  not well suited for fine-tuning structures which
  are very close to optimal solutions. Instead, the
  strength of EA is in quickly locating the high
  performance regions of vast and complex search
  spaces. Once those regions are located, it may be
  useful to apply local search heuristics to the
  high performance structures evolved by the EA.
  

21
Hierarchical classification

• Three instances of this hybridization scheme

Greedy heuristic
GA
Greedy heuristic
Initial population
Population to exploit
Initial population
GA
Tabu
Population to exploit
GA
Tabu
22
Hierarchical classification

• HCH (High-level Co-evolutionary Hybrid).
• The HCH scheme involves several
  self-contained algorithms performing a search in
  parallel, and cooperating to find an optimum.
  Intuitively, HCH will ultimately perform at least
  as well as one algorithm alone, more often
  perform better, each algorithm providing
  information to the others to help them.

23
Hierarchical classification

• An example of HCH based on GA is the island
  model

Figure 4. The island model of genetic algorithms
as an example of High-level Co-evolutionary
Hybrid.
24
Hierarchical classification

• a) The population is partitioned into
  small subpopulations
• by geographic isolation.
• c) A GA evolves each subpopulation
• b) Individuals can migrate between
  subpopulations
• d) The model is controlled by several
  parameters
• - topology
• - migration rate
• - replacement strategy
• - migration interval

25
Hierarchical classification

• e) The results of many experiment done based
  on this model show that
• the global optimum was found more often
  when migration (with cooperation) was used than
  in completely isolated cases (without
  cooperation).

26
Flat classification

• 2.2 Flat classification
• 2.2.1 Homogeneous versus Heterogeneous
• - In homogeneous hybrids, all the combined
  algorithms use the same metaheuristic. In
  general, different parameters are used for the
  algorithms.
• - In heterogeneous algorithms, different
  metaheuristics are used.

27
Flat classification

• Figure 5. High-level Co-evolutionary
  Hybridization
• HCH(heterogeneous, global,
  general). Several search
• algorithms cooperate, co-adapt,
  and co-solve a solution.

28
Flat classification

• The GRASP method may be seen as an iterated
  heterogeneous HRH hybrid, in which local search
  is repeated from a number of initial solutions
  generated by randomized greedy heuristic.

29
Flat classification

• 2.2.2 Global versus Partial
• - In global hybrids, all the algorithms
  search in the whole research space. The goal is
  here to explore the space more thoroughly.
• - In partial hybrids, the problem to be
  solved is decomposed into sub-problems, each one
  having its own search space. Then each algorithm
  is dedicated to the search in one of these
  sub-space.

30
Flat classification

• 2.2.3 Specialist versus General
• - In general hybrids, all the algorithms
  solve the same target optimization problem. All
  the above mentioned hybrids are general hybrids.
• - Specialist hybrids combine algorithms
  which solve different problems. An example of
  such a HCH approach has been developed to solve
  the quadratic assignment problem(QAP).

31
Flat classification

• Figure 6. High-level Co-evolutionary
  hybridization HCH(Global, Heterogeneous,
  Specialist). Several search algorithms solve
  different problems

32
Flat classification

• Another approach of specialist hybrid HRH
  heuristic is to use an heuristic to optimize
  another heuristic, i.e. find the optimal values
  of the parameters of the heuristic. This approach
  has been used to optimize simulated annealing by
  GA, ant colonies by GA, and a GA by a GA.

33
Implementation issues

• The structure of the taxonomy concerning
  implementation issues is shown in Figure 7.

34
Implementation issues

• Figure 7. Classification of hybrid
  metaheuristics(implementation issues).

35
Implementation issues

• Specific versus General-purpose computers
• - Application specific computers differ
  from general purpose ones in that they usually
  only solve a small range of problems, but often
  at much higher rates and lower costs. Their
  internal structure is tailored for a particular
  problem, and thus can achieve much higher
  efficiency and hardware utilization than a
  processor which must handle a wide range of
  tasks.

36
Implementation issues

• Sequential versus Parallel
• - Most of the proposed hybrid metaheuristics
  are sequential.
• - According to the size of problems,
  parallel implementations of hybrid algorithms
  have been considered. Parallel hybrids may be
  classified using the different characteristics of
  the target parallel architecture
• SIMD versus MIMD
• In SIMD (Single Instruction stream,
  Multiple Data Stream) parallel machines, the
  processors are restricted to execute the same
  program. They are very efficient in executing
  synchronized parallel algorithms that contain
  regular computations and regular data structure.

37
Implementation issues

• In parallel MIMD (Multiple Instruction
  stream, Multiple data stream), the processors are
  allowed to perform different types of
  instructions on different data. HCH hybrids based
  respectively on tabu search, simulated annealing,
  and genetic algorithms have been implemented on
  networks of transputers.
• Shared-memory versus Distributed-memory
• Shared-memory parallel architecture
  simplicity
• Distributed-memory parallel architecture
  flexible and
• 
  fault-tolerant
• 
  programming
• 
  platform

38
Implementation issues

• Homogeneous versus Heterogeneous
• - Most of massively parallel machines
  (MPP) and cluster of processors such as IBM SP/2,
  Cray T3D, and DEC Alpha-farms are composed of
  homogeneous processors.
• - Heterogeneous network of workstations
  comp up as platforms for high-performance
  computing due to the availability of powerful
  workstations and fast communication networks.
• Look at the following example

39
Implementation issues

• Figure 8. Parallel implementation of
  heterogeneous HCH algorithms.

40
Implementation issues

• 3.3 Static, Dynamic or Adaptive
• static This category represents parallel
  heuristics in which
• both the number of tasks of the
  application and the
• location of the work (tasks or
  data) are generated at
• compile time (static scheduling).
  The allocation of
• processors to tasks (or data)
  remains unchanged
• during the execution of the
  application regardless of
• the current state of the parallel
  machines. Most of
• the proposed parallel heuristics
  belong to this class.

41
The major disadvantage
When there are noticeable load or power
differences between processors, a significant
number of tasks are often idle waiting for other
tasks to complete their work.
42
Implementation issues

• dynamic To improve the performance of
  parallel static
• heuristics, dynamic
  load balancing must be
• introduced. This class
  represents heuristics
• for which the number of
  tasks is fixed at
• compile-time, but the
  location of work (tasks,
• data) is determined
  and/or changed at run-
• time. For example,
  load-balancing requirement
• can be met by a dynamic
  redistribution of
• work between
  processors.

43
Disadvantage
When the number of tasks exceeds the number of
idle nodes, multiple tasks are assigned to the
same node. Moreover, when there are more idle
nodes than tasks , some of them will not be used.
44
Implementation issues

• adaptive Parallel adaptive programs are
  parallel
• computations with a
  dynamically changing set of
• tasks. Tasks may be created
  or killed as a function
• of the load state of the
  parallel machine. A task is
• created automatically when a
  node become idle.
• When a node becomes busy,
  the task is killed.

45
A grammar for extended hybrids

• lthybrid metaheuristic gt ltdesign issuesgt
  ltimplementation issuesgt
• ltdesign issuesgt lt hierarchical gtlt flat gt
• lt hierarchical gt lt LRH gt lt LCH gt lt HRH gt
  lt HCH gt
• lt LRH gt LRH (lt metaheuristic gt(lt
  metaheuristic gt))
• lt LCH gt LCH (lt metaheuristic gt(lt
  metaheuristic gt))
• lt HRH gt HRH (lt metaheuristic gt lt
  metaheuristic gt)
• lt HCH gt HCH (lt metaheuristic gt )
• lt HCH gt HCH (lt metaheuristic gt, lt
  metaheuristic gt )
• lt flat gt ( lt nature gt, lt optimization gt,
  lt function gt )
• lt nature gt homogeneous heterogeneous
• lt optimization gt global partial

46
A grammar for extended hybrids

• lt function gt general specialist
• ltimplementation issuesgt sequential
  parallel lt schedulinggt
• lt schedulinggt static dynamic adaptive
• lt metaheuristic gt LS TS SA GA ES GP
  NN
• lt metaheuristic gt GH AC SS NM CLP
  lthybrid-metaheuristicgt

Figure 9. A grammar for extended hybridization
schemes
47
A grammar for extended hybrids

• Some hybridization examples
• HCH(HRH(GHLCH(GA(LS)))) hierarchical scheme was
  used to solve set partitioning problem.
• HRH(GHLSLCH(GA(GHLS))) scheme was used for
  traveling salesman problem.
• Three metaheuristics, GA, SA, and TS have been
  combined in a LCH(GA(HRH(SATS))) scheme to solve
  a scheduling problem.

48
Conclusion

• A taxonomy for hybrid metaheuristics has been
  presented.
• It considers solutions to design and
  implementation issues.
• Treating the two problems orthogonally is
  beneficial because it allows one to study,
  understand, classify and evaluate the algorithms
  using a well defined set of criteria.

49
Conclusion

• Hybrid metaheuristics that hybridize
  population-based metaheuristics with local search
  heuristics have been proved to be very efficient
  for large size and hard optimization problem.
• The HCH proposes natural way to efficiently
  implement algorithms on heterogeneous computer
  environment

50
References

• E-G. Talbi. A Taxonomy of Hybrid Metaheuristics.
  Journal of Combinatorial Optimization, 1-45, 1999
• V. Bachelet, P. Preux, and E-G. Talbi. Parallel
  Hybrid Meta-heuristics Application to the
  Quadratic Assignment Problem. May, 1996
• Olivier C. Martin, and Steve W. Otto. Combining
  Simulated Annealing with Local Search Heuristics.
  Annals of Operations Research, 1996
• D.E. Brown, C. L. Huntley, and A. R. Spillane. A
  parallel genetic heuristic for the quadratic
  assignment problem. In Third Int. Conf. On
  Genetic Algorithms ICGA 89, San Mateo, USA, July
  1989

51
The End

• Thank you !