Application%20of%20Genetic%20Algorithms%20and%20Neural%20Networks%20to%20the%20Solution%20of%20Inverse%20Heat%20Conduction%20Problems

1
Application of Genetic Algorithms and Neural
Networks to the Solution of Inverse Heat
Conduction Problems

• A Tutorial
• Keith A. WoodburyMechanical Engineering
  Department

2
Paper/Presentation/Programs

• Not on the CD
• Available from www.me.ua.edu/inverse

3
Overview

• Genetic Algorithms
• What are they?
• How do they work?
• Application to simple parameter estimation
• Application to Boundary Inverse Heat Conduction
  Problem

4
Overview

• Neural Networks
• What are they?
• How do they work?
• Application to simple parameter estimation
• Discussion of boundary inverse heat conduction
  problem

5
MATLAB

• Integrated environment for computation and
  visualization of results
• Simple programming language
• Optimized algorithms
• Add-in toolbox for Genetic Algorithms

6
Genetic Algorithms

• What are they?
• GAs perform a random search of a defined
  N-dimensional solution space
• GAs mimic processes in nature that led to
  evolution of higher organisms
• Natural selection (survival of the fittest)
• Reproduction
• Crossover
• Mutation
• GAs do not require any gradient information and
  therefore may be suitable for nonlinear problems

7
Genetic Algorithms

• How do they work?
• A population of genes is evaluated using a
  specified fitness measure
• The best members of the population are selected
  for reproduction to form the next generation. The
  new population is related to the old one in a
  particular way
• Random mutations occur to introduce new
  characteristics into the new generation

8
Genetic Algorithms

• Rely heavily on random processes
• A random number generator will be called
  thousands of times during a simulation
• Searches are inherently computationally intensive
• Usually will find the global max/min within the
  specified search domain

9
Genetic Algorithms

• Basic scheme
• (1)Initialize population
• (2)evaluate fitness of each member
• (3)reproduce with fittest members
• (4)introduce random mutations in new generation
• Continue (2)-(3)-(4) until prespecified number of
  generations are complete

10
Role of Forward Solver

• Provide evaluations of the candidates in the
  population
• Similar to the role in conventional inverse
  problem

11
Elitism

• Keep the best members of a generation to ensure
  that their characteristics continue to influence
  subsequent generations

12
Encoding

• Population stored as coded genes
• Binary Encoding
• Represents data as strings of binary numbers
• Useful for certain GA operations (e.g.,
  crossover)
• Real number encoding
• Represent data as arrays of real numbers
• Useful for engineering problems

13
Binary Encoding Crossover Reproduction
14
Binary Encoding

• Mutation
• Generate a random number for each chromosome
  (bit)
• If the random number is greater than a mutation
  threshold selected before the simulation, then
  flip the bit

15
Real Number Encoding

• Genes stored as arrays of real numbers
• Parents selected by sorting population best to
  worst and taking the top Nbest for random
  reproduction

16
Real Number Encoding

• Reproduction
• Weighted average of the parent arrays Ci
  wAi (1-w)Biwhere w is a random number 0 w
  1
• If sequence of arrays are relevant, use a
  crosover-like scheme on the children

17
Real Number Encoding

• Mutation
• If mutation threshold is passed, replace the
  entire array with a randomly generated one
• Introduces large changes into population

18
Real Number Encoding

• Creep
• If a creep threshold is passed, scale the
  member of the population with Ci ( 1 w
  )Ciwhere w is a random number in the range 0
  w wmax. Both the creep threshold and wmax must
  be specified before the simulation begins
• Introduces small scale changes into population

19
Simple GA Example

• Given two or more points that define a line,
  determine the best value of the intercept b and
  the slope m
• Use a least squares criterion to measure
  fitness

20
Make up some data

• gtgt b 1 m 2
• gtgt xvals 1 2 3 4 5
• gtgt yvals bones(1,5) m xvalsyvals
• 3 5 7 9 11

21
Parameters

• Npop number of members in population
• (low, high) real number pair specifying the
  domain of the search space
• Nbest number of the best members to use for
  reproduction at each new generation

22
Parameters

• Ngen total number of generations to produce
• Mut_chance mutation threshold
• Creep_chance creep threshold
• Creep_amount parameter wmax

23
Parameters

• Npop 100
• (low, high) (-5, 5)
• Nbest 10
• Ngen 100

24
SimpleGA Results (exact data)
25
SimpleGA Convergence History
26
SimpleGA Results (1 noise)
27
SimpleGA Results (10 noise)
28
SimpleGA 10 noise
29
Heat function estimation

• Each member of population is an array of Nunknown
  values representing the piecewise constant heat
  flux components
• Discrete Duhamels Summation used to compute the
  response of the 1-D domain

30
Make up some data

• Use Duhamels summation with ?t 0.001
• Assume classic triangular heat flux

31
Data
32
Two data sets

• Easy Problem large ?t
• Choose every third point from the generated set
• Harder Problem small ?t
• Use all the data from the generated set

33
GA program modifications

• Let Ngen, mut_chance, creep_chance, and
  creep_amount be vectors
• Facilitates dynamic strategy
• Example Ngen 100 200 mut_chance
  0.7 0.5 means let mut_chance 0.7 for 100
  generations and then let mut_chance 0.5 until
  200 generations

34
GA Program Modifications

• After completion of each pass of the Ngen array,
  redefine (low,high) based on (min,max) of the
  best member of the population
• Nelite 5

35
Easy Problem

• ?t 0.18
• First try, let
• Npop 100
• (low, high) (-1, 1)
• Nbest 10
• Ngen 100

36
Easy Problem

• Npop 100, Nbest 10, Ngen 100

37
Easy Problem

• Npop 100, Nbest 10, Ngen 100

38
Easy Problem

• Npop 100, Nbest 10, Ngen 100

39
Easy Problem another try

• Use variable parameter strategy
• Nbest 20
• Ngen 200 350 500 650 750
• mut_chance 0.9 0.7 0.5 0.3 0.1
• creep_chance 0.9 0.9 0.9 0.9 0.9
• creep_amount 0.7 0.5 0.3 0.1 0.05

40
Easy Problem another try
41
Easy Problem another try
42
Easy Problem another try
43
Hard Problem

• has small time step data ?t 0.06
• Use same parameters as last
• Nbest 20
• Ngen 200 350 500 650 750
• mut_chance 0.9 0.7 0.5 0.3 0.1
• creep_chance 0.9 0.9 0.9 0.9 0.9
• creep_amount 0.7 0.5 0.3 0.1 0.05

44
Hard Problem
45
Hard Problem
46
Hard Problem
47
Hard Problem

• Whats wrong?
• Ill-posedness of the problem is apparent as ?t
  becomes small.
• Solution
• Add a Tikhonov regularizing term to the objective
  function

48
Hard Problem

• With ?1 1.e-3

49
Hard Problem

• With ?1 1.e-3

50
Hard Problem

• With ?1 1.e-3

51
Genetic Algorithms- Conclusions

• GAs are a random search procedure
• Domain of solution must be known
• GAs are computationally intensive
• GAs can be applied to ill-posed problems but
  cannot by-pass the ill-posedness of the problem
• Selection of solution parameters for GAs is
  important for successful simulation

52
Neural Networks

• What are they?
• Collection of neurons interconnected with weights
• Intended to mimic the massively parallel
  operations of the human brain
• Act as interpolative functions for given set of
  facts

53
Neural Networks

• How do they work?
• The network is trained with a set of known facts
  that cover the solution space
• During the training the weights in the network
  are adjusted until the correct answer is given
  for all the facts in the training set
• After training, the weights are fixed and the
  network answers questions not in the training
  data.
• These answers are consistent with the training
  data

54
Neural Networks

• The neural network learns the relationship
  between the inputs and outputs by adjustment of
  the weights in the network
• When confronted with facts not in the training
  set, the weights and activation functions act to
  compute a result consistent with the training data

55
Neural Networks

• Concurrent NNs accept all inputs at once
• Recurrent or dynamic NNs accept input
  sequentially and may have one or more outputs fed
  back to input
• We consider only concurrent NNs

56
Role of Forward Solver

• Provide large number of (input,output) data sets
  for training

57
Neural Networks
58
Neurons
Activation function
summation
59
MATLAB Toolbox

• Facilitates easy construction, training and use
  of NNs
• Concurrent and recurrent networks
• Linear, tansig, and logsig activation functions
• Variety of training algorithms
• Backpropagation
• Descent methods (CG, Steepest Descent)

60
Simple Parameter Estimation

• Given a number of points on a line, determine the
  slope (m) and intercept (b) of the line
• Fix number of points at 6
• Restrict domain 0 x 1
• Restrict range of b and m to 0, 1

61
Simple Example

• First approach let the six values of x be fixed
  at 0, 0.2, 0.4, 0.6, 0.8, and 1.0
• Inputs to the network will be the six values of y
  corresponding to these
• Outputs of the network will be the slope m and
  intercept b

62
Simple Example

• Training data
• 20 columns of 6 rows of y data

Columns 1 through 8 0 0.2500
0.5000 0.7500 1.0000 1.0000 0.7500
0.5000 0.2000 0.4000 0.6000 0.8000
1.0000 1.0000 0.8000 0.6000 0.4000
0.5500 0.7000 0.8500 1.0000 1.0000
0.8500 0.7000 0.6000 0.7000 0.8000
0.9000 1.0000 1.0000 0.9000
0.8000 0.8000 0.8500 0.9000 0.9500
1.0000 1.0000 0.9500 0.9000 1.0000
1.0000 1.0000 1.0000 1.0000 1.0000
1.0000 1.0000
63
Simple Example

• Network1
• 6 inputs linear activation function
• 12 neurons in hidden layer
• Use tansig activation function
• 2 outputs linear activation function
• trained until SSE lt 10-8

64
Test Input Data

• input_data1
• 0.9000 0.1000 0.9000 0.3000
  0.5000 0.3000 0.7000 0.7000
• 1.0400 0.2800 0.9200 0.4000
  0.5600 0.4400 0.7600 0.8800
• 1.1800 0.4600 0.9400 0.5000
  0.6200 0.5800 0.8200 1.0600
• 1.3200 0.6400 0.9600 0.6000
  0.6800 0.7200 0.8800 1.2400
• 1.4600 0.8200 0.9800 0.7000
  0.7400 0.8600 0.9400 1.4200
• 1.6000 1.0000 1.0000 0.8000
  0.8000 1.0000 1.0000 1.6000
• output_data1
• 0.9000 0.1000 0.9000 0.3000
  0.5000 0.3000 0.7000 0.7000
• 0.7000 0.9000 0.1000 0.5000
  0.3000 0.7000 0.3000 0.9000

65
Simple Example

• Network1Test data1 results
• b m bNN
  mNN
• 0.9000 0.7000 0.8973 0.7236
• 0.1000 0.9000 0.0997 0.9006
• 0.9000 0.1000 0.9002 0.0998
• 0.3000 0.5000 0.3296 0.5356
• 0.5000 0.3000 0.5503 0.3231
• 0.3000 0.7000 0.3001 0.6999
• 0.7000 0.3000 0.7000 0.3001
• 0.7000 0.9000 0.7269 0.8882

66
Network2

• Increase number of neurons in hidden layer to 24
• Train until SSE lt 10-15
• Test data1 results b m
  bNN mNN
• 0.9000 0.7000 0.8978 0.7102
• 0.1000 0.9000 0.0915 0.9314
• 0.9000 0.1000 0.9017 0.0920
• 0.3000 0.5000 0.2948 0.5799
• 0.5000 0.3000 0.4929 0.3575
• 0.3000 0.7000 0.3017 0.6934
• 0.7000 0.3000 0.6993 0.3032
• 0.7000 0.9000 0.7008 0.8976

67
Network3

• Add a second hidden layer with 24 neurons
• Train until SSE lt 10-18
• Test data1 results b m
  bNN mNN
• 0.9000 0.7000 0.8882 0.7018
• 0.1000 0.9000 0.1171 0.9067
• 0.9000 0.1000 0.8918 0.1002
• 0.3000 0.5000 0.3080 0.5084
• 0.5000 0.3000 0.5033 0.3183
• 0.3000 0.7000 0.2937 0.6999
• 0.7000 0.3000 0.7039 0.2996
• 0.7000 0.9000 0.7026 0.9180

68
Network Design

• First add more neurons in each layer
• Add more hidden layers if necessary

69
Simple Example Second Approach

• Add the 6 values of x to the input (total of 12
  inputs)
• Network4
• 24 neurons in hidden layer
• Trained until SSE lt 10-12

70
Network4

• Test data1 results
• b m bNN mNN
• 0.9000 0.7000 0.8972 0.7171
• 0.1000 0.9000 0.1016 0.9081
• 0.9000 0.1000 0.9002 0.0979
• 0.3000 0.5000 0.2940 0.5059
• 0.5000 0.3000 0.5115 0.3013
• 0.3000 0.7000 0.2996 0.6980
• 0.7000 0.3000 0.7000 0.3009
• 0.7000 0.9000 0.7108 0.8803

71
Network4

• Try with x values not in the training data (x
  0.1, .3, .45, .55, .7, .9 )
• b m bNN
  mNN
• 0.9000 0.7000 0.9148 0.3622
• 0.1000 0.9000 0.1676 0.4901
• 0.9000 0.1000 0.9169 -0.1488
• 0.3000 0.5000 0.3834 0.1632
• 0.5000 0.3000 0.5851 0.0109
• 0.3000 0.7000 0.3705 0.3338
• 0.7000 0.3000 0.7426 0.0139
• 0.7000 0.9000 0.7284 0.5163

72
NNs in Inverse Heat Conduction

• Two possibilities whole domain and sequential
• Concurrent networks offer best possibility of
  solving the whole domain problem (Krejsa, et al
  1999)

73
NNs in Inverse Heat Conduction

• Training data
• Must cover domain and range of possible
  inputs/outputs
• Use forward solver to supply solutions to many
  standard problems (linear, constant, triangular
  heat flux inputs)

74
NNs in Inverse Heat Conduction

• Ill-posedness?
• One possibility train network with noisy data

75
NNs in Inverse Heat Conduction

• Krejsa, et al (1999) concluded that
• Radial basis functions and cascade correlation
  networks offer a better possibility for solution
  of the whole domain problem than standard
  backpropagation networks

76
Neural Networks - Conclusions

• NNs offer possibility of solution of parameter
  estimation and inverse problems
• Proper design of the network and training set is
  essential for successful application

77
References

• M. Raudensky, K. A. Woodbury, J. Kral, and T.
  Brezina,, Genetic Algorithm in Solution of
  Inverse Heat Conduction Problems, Numerical Heat
  Transfer, Part B Fundamentals, Vol 28, no 3,
  Oct.-Nov. 1995, pp. 293-306.
• J. Krejsa, K. A. Woodbury, J. D. Ratliff., and M.
  Raudensky, Assessment of Strategies and
  Potential for Neural Networks in the IHCP,
  Inverse Problems in Engineering, Vol 7, n 3, pp.
  197-213. (1999)