Sensitivity Analysis

1
Sensitivity Analysis

• Multi-disciplinary Design Optimization
• Centre for Aerospace Systems Design Engineering
• Department of Aerospace Engineering
• Indian Institute of Technology
• Mumbai 400 076

2
Sensitivity Analysis

• Engineering Analysis for MDO is usually
  computationally intensive
• Evolutionary algorithms for optimization require
  too many function calls (analysis)
• Efficient, accurate evaluation of gradients is
  required by gradient based optimizer
• Sensitivity Analysis (generating gradients) is an
  important component in MDO

3
Sensitivity Analysis

• Sensitivity Analysis
• Derivative of output w.r.t. input
• Gradient of output w.r.t. inputs
• Jacobian of outputs w.r.t. inputs

4
Sensitivity Analysis

• Methods for sensitivity estimation
• Finite Difference Method
• Analytical
• Direct
• Adjoint Continuous, Discrete
• Complex Variables Method
• Automatic Differentiation
• ADIFOR
• ADIC
• Global Sensitivity Equation (GSE)

5
Finite Difference Method
6
Finite Difference Methods

• Forward, backward or central differences may be
  used
• Derivative of yj w.r.t. xi by forward difference
  is given by
• Note all other elements of vector x are held
  fixed at some value
• N derivatives will require N1 function
  evaluations

7
Finite Difference Methods

• Advantages
• Almost nil effort to implement
• Most optimizers support finding derivatives by
  finite difference from within (requiring no
  effort from user)
• Well behaved with smooth functions that are
  evaluated using non-iterative methods

8
Finite Difference Methods

• Disadvantages
• Noisy functions yield poor results. Most
  iterative analysis with termination criteria are
  noisy. Most engineering analysis are non-linear
  and iterative in nature.
• Accuracy sensitive to step size. 1 pp3 pp8
• N derivatives require N1 analysis.
  Computationally intensive. 2 pp 87

9
Noisy Function, Step Size
f


y
Step size? Large Averaging Small round off in
df
10
An Example

• f(y, z) y0.1 z 0.2 , ?f/?y 0.1 y -0.9
  z 0.2
• y 0.2, z 0.3
• f 0.669156, Analytical ?f/?y 0.334578
• Error in FD derivative
• dy ?f/?y by FD err
• 0.04000000 0.307801 -8.003
• 0.00250000 0.332713 -0.557
• 0.00015625 0.334167 -0.123
• 0.00000977 0.329590 -1.491
• 0.00000061 0.292969 -12.436

11
Check Derivative Routine

• f(xd) f(x) d f od2
• g(d) f(xd) f(x) d f od2
• g(d) / g(d/2) od2 / o(d/2)2
• Use function and derivative routines to compute
  g(d) / g(d/2). If this value is of the order of
  4 then the 2 routines are consistent

12
Check Derivative Routine

• x1 0.2 x2 0.3
• Function f 0.669156, Analytical ?f/?x1
  0.334578
• Verification of FD Derivatives
• FD Step ?f/?x1 g(d)/g(d/2) d0.03
• 0.010000 0.327283E00 7.659
• 0.002500 0.332713E00 4.213
• 0.000625 0.334072E00 3.920
• 0.000156 0.334167E00 3.903
• 0.000039 0.334167E00 3.903
• 0.000010 0.329590E00 5.406
• 0.000002 0.317383E00
  -1.077
• 0.000001 0.292969E00 1.360
• 0.334578E00 3.989 analytical
• derivative

13
Analytical Derivatives
14
Derivative of Response of an Analysis

• Evaluate f(x),
• Subject to h(x) 0
• x, f, h are all vectors
• x ? ?nx f ? ?nf h ? ?nz
• nx gt nz
• for any arbitrary x, h(x) ? 0
• Only (nx nz) values of x can be
• prescribed
• nz values of x get determined
• by h(x) 0

h(x) ? 0 violates some Conservation. It is not
OK to look at f(x) when x violates Physics.
15
Derivative of Response of an Analysis
16
Analysis

• Analysis, h(y,z) 0
• Input y design variable(s)
• Output z state variable(s)
• Analysis for non-linear system requires
• Value of y and initial guess for z
• evaluator for h in terms of y z
• iterator to update z, so that h(y,z) is driven to
  0.
• Convergence criteria to arrive at z(y)

17
Analysis
18
Response of Analysis

• Evaluate f(y, z)
• Subject to, h(y, z) 0
• Note
• y alone is design variable
• z is output of analysis, zy
• df/dy is required ? total derivative

19
Analytical Derivatives Across Analysis

• Sensitivity required, df/dy ? (Let y, z, f, h
  be all scalars)
• df/dy ?f/?y ?f/?z dz/dy
• dh ?h/?y dy ?h/?z dz 0
• dz/dy - (?h/?z)-1 ?h/?y
• df/dy ?f/?y - ?f/?z (?h/?z)-1 ?h/?y
• Requires all partial derivatives at (y, z)
• Analytical expressions for partial derivatives?

20
Analytical Derivatives Across Analysis

• Sensitivity Required
• df/dy ?f/?y - ?f/?z (?h/?z)-1 ?h/?y
• Direct Method
• df/dy ?f/?y - ?f/?z (?h/?z)-1 ?h/?y
• OR
• Adjoint Method
• df/dy ?f/?y - ?f/?z (?h/?z)-1 ?h/?y
• Difference!
• In the effort to compute bracketed quantities
• (solution of linear algebraic equations)
• when f and y are vectors

21
An Example

• f(y, z) y0.1 z 0.2
• ?f/?y 0.1 y -0.9 z 0.2 , ?f/?z
  0.2 y 0.1 z -0.8
• h(y, z) y0.3 z 0.4 - 1.0
• ?h/?y 0.3 y -0.7 z 0.4, ?h/?z 0.4 y
  0.3 z -0.6
• Y 0.2, Analysis No of iters 62, z
  3.343695
• ?f/?y 0.541899 ?f/?z 0.064826
• ?h/?y 1.499999 ?h/?z 0.119628
• df/dy ?f/?y - ?f/?z (?h/?z)-1 ?h/?y -
  0.27094

22
An Example

• Comparison With FD Across Analysis
• y 0.2, ? z 3.343695
• f 1.08380 Analytical deriv, df/dy -
  0.27094
• dy df/dy by FD
• 0.00400000 -.26813149
• 0.00025000 -.27084351
• 0.00001563 -.26702881
• 0.00000391 -.21362303
• 0.00000024 -.48828122
• 0.00000006 1.95312488

23
An Example - Timings

• Let time for evaluation T
• ie. evaluation of h, f and their analytical
  derivatives
• FD across Analysis
• 2 (Niter T T) 2 T (Niter 1)
• (Analysis evaluation of f) to be done twice
• Analytical Derivative
• Niter T 4 T T T (Niter 5)
• 1 analysis 4 derivatives f
• Note Niter 62 in this case

24
When f, h, y, z Are All Vectors

• Direct Method

25
When f, h, y, z Are All Vectors

• Adjoint Method

26
Analytical Derivatives

• Direct and Adjoint Methods require partial
  derivatives of f and h
• Direct method requires solution of nznz system
  for ny RHS vectors
• Adjoint method requires solution of nznz syetm
  for nf RHS vector
• The nznz matrix is often the same as used in
  analysis

27
Direct / Adjoint Methods

• Based on Control Theory Concept

28
Introduction

• Proposed by Antony Jameson (Stanford)
• Aerodynamic Design via Control Theory Journal
  of Scientific Computing, vol. 3, 233-260, 1988.
• Optimization is treated as a control problem with
  constraints as the control variables

Continuous Adjoint Method
Discrete Adjoint Equations-1
Continuous Adjoint Equations
Continuous Field Equations h(y,z)
Discrete Field Equations
Discrete Adjoint Equations-2
Discrete Adjoint Method
29
Basic Formulation
(Adjoint Equation)
If ? is such that
then dI G. dy where

• Unlike FDM, solution of h(y,z)0 is not required
  for each design variable

30
Quasi 1D Nozzle Flow
Objective function
31
Quasi 1D Nozzle Flow(contd.)
Equating terms with dU, Adjoint equation is
Adjoint boundary conditions

• Observations
• Similar procedure for higher dimensional flows
• Form of adjoint equation is strongly linked to
  the field equation
• Adjoint equations may be solved using similar
  discretization and numerical solution strategies
  of the field equations

32
Algorithm
Calculation of ?f/?y
Solution of Field Eq. h(y,z)0
Flow variables (z)
Solution of Adjoint Eq.
Lagrange Multipliers (?)
Calculation of ?h/ ?y (?h/ ?zi)(?zi/?aj)(?aj/?y
)
Calculation of Gradients (G)
?ai/?y
Grid Generator
a Grid variables (Cell areas/lengths)
33
Road Ahead

• Groups actively involved in further research
• Stanford University (Jameson et al)1,5,6,7
• Oxford University (Mike Giles)8
• Virginia Tech. (Dr. Eugene Cliff)3
• Old Dominian University (Baysal et al)
• Applications
• Extension to unstructured grids 2
• Optimization of a supersonic aircraft
  configuration9
• Further areas of research
• A new reduced formulation for adjoint methods
  proposed by Jameson7 to eliminate grid
  sensitivity of gradients.

34
Complex Step Method
35
Complex Variable Method

• For f(x) ? R, it can be shown using
  Cauchy-Riemann equality that
• As no subtraction is involved, step size can be
  as small as the machine precision allows
• It has been found that usually a step size lt 10-8
  gives near analytic solutions2
• First proposed by Lyness Moler1 in 1967

36
CVM An example
37
CVM Implementation
Declare all the variables as complex type

• All the algorithms can be broken into basic
  operations
• Operations that require redefinition (usually)
• Relational Operators
• Arithmetic functions operators
• FORTRAN-90 intrinsically supports most of the
  complex operations
• Two options for CVM implementation
• Source modification
• Operator Overloading

Redefine all the functions with complex
definitions
Modification of the I/O statements
Add complex step (h) to the desired variable
Gradient Calculation
38
CVM General Remarks

• Advantages
• Experimentation with step size not required
• Easy implementation
• Disadvantages
• Separate simulations for each gradient required
• Implementation of CVM as an automatic
  differentiation tool3
• Groups working in this area
• Squire Trapp4
• Anderson Newman 5

39
An Example

• f(y, z) y0.1 z 0.2 , ?f/?y 0.1 y -0.9
  z 0.2
• y 0.2, z 0.3
• Comparison with CVM
• f complex f anal df/dx1
  dx1 comp df/dx1
• 0.669156 0.670339 0.3345779 0.4000E-01
  0.3308477
• 0.669156 0.669168 0.3345779 0.4000E-02
  0.3345398
• 0.669156 0.669156 0.3345779 0.4000E-04
  0.3345779
• 0.669156 0.669156 0.3345779 0.4000E-05
  0.3345779
• 0.669156 0.669156 0.3345779 0.4000E-06
  0.3345779
• 0.669156 0.669156 0.3345779 0.4000E-08
  0.3345779
• 0.669156 0.669156 0.3345779 0.4000E-10
  0.3345779

40
An Example Timing Issues

• Complex arithmetic takes less than 4 times real
  arithmetic
• Small enough value of h can yield both f and
  ?f/?y in single execution

41
Automatic Differentiation
42
User Supplied Gradients
43
User Supplied Gradients
44
User Supplied Gradients
45
Gradients by ADIFOR /ADIC
46
How ADI uses chain rule?

• function1 ( x, y, z )
• function2 ( x, y, w, r )
• function3 ( w, r )
• function4 ( w, r, z )
• We are interested in
• ?z/?x and ?z/?y

• For function3 we can write
• ?r/?w
• If we want to use ?r/?w in
• calculation of ?r/?x, we need
• to use chain rule.
• ?r/?x ?r/?w ?w/?x
• In general we can write
• r ?r/?w w

47
How to use ADI generated code?

• function1 ( x, y, z )
• function2 ( x, y, w, r )
• function3 ( w, r )
• function4 ( w, r, z )

• f1 ( x, y, z, x, y, z )
• f2 ( x, y, w, r, x, y, w, r )
• f3 ( w, r, w, r )
• f4 ( w, r, z, w, r, z )

New function f1 evaluates the values and
directional gradient specified by direction (x,
y) z ?z/?x ?z/?y x yT To evaluate the
gradient vector we do the above evaluation in
directions (1, 0) and (0, 1)
48
ADI Sample Code

• To evaluate following function
• function abc ( y, z, w )
• w -y / (z z z)
• end
• ADI software usually breaks
• down the complex expressions
• into simple operations. This
• requires introduction of
• temporary variables to store
• intermediate results

• function abc ( y, z, w )
• t1 -y
• t2 z z
• t3 t2 z
• w t1 / t3
• end
• ADI will use the above code
• to use symbolic differentiation
• to calculate gradients
• Forward Mode
• Reverse Mode

49
ADI Forward Mode

• Uses successive chain rule to derive the
  gradients
• abc ( y, z, w )
• t1 -y
• t2 z z
• t3 t2 z
• w t1 / t3

• abc ( y, y, z, z, w, w )
• t1 -y
• t1 -y
• t2 z z
• t2 z z z z
• t3 t2 z
• t3 t2 z z t2
• w t1 / t3
• w ( t1 t3 w ) / t3

50
ADI Reverse Mode

• Set of operations can be looked as series of
  operators working on input variables and
  generating new variables

• abc ( y, z, w )
• t1 -y
• t2 z z
• t3 t2 z
• w t1 / t3

w W( T3( T2( T1( y, z ) ) ) w ?W/?T3
?T3/?T2 ?T2/?T1 (?T1/?y y ?T1/?z z)
If gradient is calculated this way, it is
actually in the reverse direction of the basic
computation. Hence the term reverse mode.
51
ADI Reverse Mode

• Basic Rules
• y f(s)
• sbar sbar ?y/?s ybar
• y f(s, t)
• sbar sbar ?y/?s ybar
• tbar tbar ?y/?t ybar
• abc ( y, z, w )
• t1 -y
• t2 z z
• t3 t2 z
• w t1 / t3

• abc ( y, y, z, z, w, w )
• t1 - y
• t2 z z
• t3 t2 z
• w t1 / t3
• t1bar 1 / t3
• t3bar -t1 / (t3 t3)
• t2bar t3bar z
• zbar t3bar t2
• zbar zbar t2bar z
• zbar zbar t2bar z
• ybar - t1bar
• w ybar y zbar z

52
ADI generated code

• FORWARD MODE
• t1 - y
• t1 - y
• t2 z z
• t2 z z z z
• t3 t2 z
• t3 t2 z t2 z
• w t1 / t3
• w (t1 - t3 w) / t3

• REVERSE MODE
• t1 - y
• t2 z z
• t3 t2 z
• w t1 / t3
• t1bar 1 / t3
• t3bar -t1 / (t3 t3)
• t2bar t3bar z
• zbar t3bar t2
• zbar zbar t2bar z
• zbar zbar t2bar z
• ybar - t1bar
• w ybar y zbar z

53
ADI Comments

• The forward mode can be used to produce the
  partial derivatives of all dependent variables
  w.r.t. a single independent variable time
  proportional to the evaluation of F
• The reverse mode can be used to produce the
  partial derivatives of a single dependent
  variable w.r.t. all independent variables in time
  proportional to the evaluation of F
• ? y1, , yn ? F (? x1, , xm ?)
• For values of n ?? m, the forward mode is more
  efficient
• For values of m ?? n, the reverse mode is more
  efficient

54
Global Sensitivity Equation
55
Global Sensitivity (GS)

• Interest is in Multi-Disciplinary Analysis (MDA)
• MDA are coupled analysis
• GS -gt
• GSE requires running
• Analysis to convergence
• twice

56
Local Sensitivity (LS)

• LS -gt ?? -gt

57
GS LS
58
Global Sensitivity Equation

• Requires sensitivity of disciplinary analysis
  only
• Can be computed after MDA converges

59
Time Savings

• Suppose
• Analysis-1 and Analysis-2 require n1 and n2
  iterations respectively,
• T is the time per evaluation for both the
  analyses, and
• N iterations are required for MDA convergence,
  then
• Time for FD across MDA
• 2N(n1 n2)T
• Time using GSE
• (N1)(n1 n2)T

60
Global Sensitivity Equations

• First introduced by Sobieski1 in 1990
• Successful implementations
• HSCT project by Anthony Giunta2
• Supersonic Transport Aircraft by Barthelemy
  Dovi3
• Forward swept wing by Kapania Eldred4

61
Sensitivity Landscape
62
General References

• James Newman, et. al. Overview of Sensitivity
  Analysis and Shape Optimization for Complex
  Aerodynamic Configurations, J A/C, Vol 36, No.
  1, Jan-Feb 1999

63
References FDM

• Hicks Henne, Wing design by numerical
  optimization, Journal of Aircraft vol. 15, no.
  7, July 1978.

64
References Complex Variable

• Lyness, J N Moler C B, Numerical
  differentiation of analytic functions,SIAM J.
  Numerical Analysis, Vol. 4, 1967, pp. 202-210.
• Joaquim Martins, Sensitivity Analysis, Internal
  Report, Aero. Astro. Dept. Stanford University.
• Martins, Sturdza, Alonso, The Connection between
  the Complex-Step Derivative Approximation and
  Algorithmic Differentiation, Proceedings of the
  39th Aerospace Sciences Meeting and Exhibit,
  Reno, NV, Jan. 2001.
• Squire, W. and Trapp, G., Using Complex
  Variables to Estimate Derivatives of Real
  Functions, SIAM Review, Vol. 10, No. 1, 1998 pp
  110-112.
• Anderson, Newman et. Al., Sensitivity Analysis
  for the Navier-Stokes Equations on Unstructured
  Meshes Using Complex Variables, AIAA Journal,
  Vol. 39, No. 1, 2000 p. 56-62.

65
References Adjoint Method

• Antony Jameson Aerodynamic Design via control
  theory, J. of scientific Computing Vol. 3
  233-260, 1988.
• Anderson, Venkatakrishnan, Aerodynamic Design
  Optimization on Unstructured Grids with a
  Continuous Adjoint Formulation, AIAA 97-0643.
• Lei Xei, Gradient-based Optimum Aerodynamic
  Design using adjoint method, PhD Thesis,
  Virginia Polytechnic Institute and state
  university, Aerospace Engineering, 2002.
• Dadone, Grossman, Fast convergence of inviscid
  fluid dynamic design problems, Computers and
  fluids Vol. 32, 607-627, 2003.
• Antony Jameson, Introduction to Adjoint
  Methods, Lectures in Von Karmen Institute of
  Fluid Mechanics, Feb. 2003.
• Siva Nadarajah Antony Jameson, A Comparison of
  the Continuous Discrete Adjoint Approach to
  Automatic Aerodynamic Optimization,
  AIAA-2000-0667
• Antony Jameson Sangho Kim, Reduction of the
  Adjoint Gradient Formula for Aerodynamic Shape
  Optimization Problems, AIAA Journal, Vol. 41,
  No. 11, Nov. 2003.
• M.B. Giles and N.A. Pierce. Improved lift and
  drag estimates using adjoint Euler equations'.
  AIAA Paper 99-3293, 1999.
• Alonso et al,Aerodynamic Shape Optimization fo
  Supersonic Aircraft configurations via an Adjoint
  Formulation on Parallel Computers, Computers and
  Fluids, Vol. 28. No. 4, 1999, pp. 675-700.

66
References Automatic Differentiation

• Griewank A., On Automatic Differentiation, in
  Mathematical Programming Recent developments and
  Applications, M. Iri and K. Tanabe, ets., Kluwer
  Academic Publishers, 1989, pp. 83-108.
• Bischof C., Corliss G., Green L., Griewank A.,
  Haigler K. and Newman P., Automatic
  Differentation of Advanced CFD Codes for
  Multidisciplinary Design, Journal of Computing
  systems in Engineering, Vol. 3., 1992, pp.
  625-638.
• Jueses David, A Taxonomy of Automatic
  Differentiation Tools, Proceedings of the
  Workshop on Automatic Differentiation of
  Algorithms Theory, Implementation and
  Application, SIAM, 1991, pp. 315-330.
• Computational Differentiation Technical Reports
  http//www-fp.mcs.anl.gov/autodiff/tech_reports.h
  tml

67
References GSE

• Sobieszczanski-Sobieski, J., Sensitivity of
  Complex, Internally Coupled Systems, AIAA J.,
  vol. 28, no. 1, Jan. 1990, pp. 153-160.
• Anthony Giunta, Sensitivity Analysis for Coupled
  Aero-structural Systems, NASA/TM-1999-209367.
• Barthelemy et. al., Supersonic Transport Wing
  Minimum Weight Design Integrating Aerodynamics
  and Structures, J. Aircraft, 31(2), 330-338
  (1994).
• Kapania, R. K. and Eldred, L. B., Sensitivity
  Analysis of Wing Aeroelastic Response, J.
  Aircraft, 30(4), 496-504 (1993).