An Introduction to Optimization Theory

1
An Introduction to Optimization Theory
2
Outline

• Introduction
• Unconstrained optimization problem
• Constrained optimization problem

3
Introduction

• Mathematically speaking, optimization is the
  minimization of a objective function subject to
  constraints on its variables. Mathematically, we
  have

4
Introduction
5
Introduction-Linear regression
6
Introduction-Battery charger
7
Unconstrained optimization problem

• Definition for unconstrained optimization problem

8
Unconstrained optimization problem
9
Gradient descent algorithm
10
Gradient descent algorithm

• Gradient descent algorithm may be trapped into
  the local extreme instead of the global extreme

11
Gradient descent algorithm

• Methodology for choosing suitable step size ak
  ---- Steepest descent algorithm

12
Gradient descent algorithm
13
Gradient descent algorithm

• Steepest descent algorithm with quadratic cost
  function

14
Gradient descent algorithm
Update equation
15
Newton method

• Summary for Newton method

16
Newton method
17
Newton method

• Procedure for Newton method

18
Quasi-Newton method
19
Quasi-Newton method

• What properties of F(x(k))-1 should it mimic ?
• 1. Hk should be a symmetric matrix
• 2. Hk should with secant property

20
Quasi-Newton method

• Typical approaches for Quasi-Newton method
• 1. Rank-one formula
• 2. DFP algorithm
• 3. BFGS algorithm (L-BFGS , L indicates
  limited-memory)

21
Constrained optimization problem

• Definition for constrained optimization problem

22
Problems with equality constraints ---- Lagrange
multiplier
23
Problems with equality constraints ---- Lagrange
multiplier
24
Problems with equality constraints ---- Lagrange
multiplier
25
Problems with equality constraints ---- Lagrange
multiplier

• Suppose x is a local minimizer

26
(No Transcript)
27
Karush-Kuhn-Tucker condition (KKT)

• From now on, we will consider the following
  problem

28
Karush-Kuhn-Tucker condition (KKT)
Note that
29
Image statistics Image enhancement

• Illustration for gradient descent with projection
  

30
Useful Matlab introductions for optimization

• Useful instructions included in Matlab for
  optimization
• 1. fminunc Solver for unconstrained
  optimization problems
• 2. fmincon Solver for constrained
  optimization problems
• 3. linprog Solver for linear programming
  problems
• 4. quadprog Solver for quadratic programming
  problems