CLASSICAL OPTIMIZATION THEORY

1
CLASSICAL OPTIMIZATION THEORY
Quadratic forms
Let
be a n-vector.
2
Let A ( aij) be a nn symmetric matrix.
We define the kth order principal minor as the
k?k determinant
3
Then the quadratic form
(or the matrix A) is called
Positive semi definite if XTAX 0 for all
X. Positive definite if XTAX gt 0 for all X ?
0. Negative semi definite if XTAX 0 for all
X. Negative definite if XTAX lt 0 for all X ? 0.
4
A necessary and sufficient condition for A (or
Q(X)) to be
Positive definite (positive semi definite) is
that all the n principal minors of A are gt 0
( 0).
Negative definite ( negative semi definite) if
kth principal minor of A has the sign of (-1)k ,
k1,2,,n (kth principal minor of A is zero or
has the sign of (-1)k , k1,2,,n )
5
Let f (X)f (x1 ,x2 ,,xn ) be a real-valued
function of the n variables x1 ,x2 ,,xn (we
assume that f (X) is at least twice
differentiable ).
A point X0 is said to be a local maximum of f (X)
if there exists an e gt 0 such that
for all
Here
and
6
X0 is said to be a local minimum of f (X) if
there exists an e gt 0 such that
for all
X0 is called an absolute maximum or global
maximum of f (X) if
X0 is called an absolute minimum or global
minimum of f (X) if
7
Theorem A necessary condition for X0 to be an
optimum point of f (X) is that
(that is all the first order partial derivatives
are zero at X0.)
Definition
A point X0 for which
is called a stationary point of f (X) (potential
candidate for local maximum or local minimum).
8
Theorem
Let X0 be a stationary point of f (X). A
sufficient condition for X0 to be a local minimum
of f (X) is that the Hessian matrix H(X0) is
positive definite local maximum of f (X) is that
the Hessian matrix H(X0) is negative definite.
9
Here H(X) is the nn matrix whose ith row are
the partial derivates of
(j 1,2,..n)
with respect to xi.
(i 1,2,..n)
i.e. H(X)
10
Problem 3 set 20.1A page 705 Find the stationary
points of the function f (x1,x2,x3) 2x1x2x3
4x1x3 2x2x3 x12 x22 x32 2x1 - 4x2
4x3 And hence find the extrema of f (X)

11
All these above equations 0 give
(1) (2) (3)
12
Substituting in (3) for x2, we get 2x1 x1x3
x12x3 2x1 2 x3 x1x3 x3 -2 Or
x1x3 (2 x1) 0 Therefore x1 0 or x3 0
or x1 2
13

• Case (i) x1 0
• implies x2x3 2x3 1 (4)
• (2) implies x2 x3 2 (5)
• (3) implies -x2x3 -2 same as (5)
• (4) using (5) gives
• x3(2 x3) 2x3 1
• or x32 1

i.e. x3
14
Sub case (i) x3 1 gives (using (5) ) x2 3
(ii) x3 -1 gives (using (5) ) x2 1
Therefore, we get 2 stationary points (0,3,1),
(0,1,-1).
15
Case (ii) x3 0 (1) gives x1 1
(2) gives x2 2 For these x1, x2 LHS of (3)
x1x2 - 2x1 - x2 -2 RHS Therefore, we get
the stationary point (1,2,0)
16
Case (iii) x1 2 (1) gives x2x3 2x3
-1 (6) (2) gives x2 x3 2
(7) (3) gives x2 x3 2
same as (7)
(6) Using (7) gives x3(2 x3) 2x3
-1 i.e. x32 1, therefore x3 1
17
sub case (i) x3 1 gives (using (5)) x2 1
(ii) x3 -1, gives (using (5) x2 3
Therefore, we get two stationary points
(2,1,1), (2,3,-1).
Hessian matrix
18
(No Transcript)
19
Definition A function f(X)f(x1,x2,xn) of n
variables is said to be convex if for each pair
of points X,Y on the graph, the line segment
joining these two points lies entirely above or
on the graph. i.e. f((1-?)X ?Y)
(1-?)f(X) ?f(Y) for all ?
such that 0 lt ? lt 1.
20
f is said to be strictly convex if for each pair
of points X,Y on the graph, f ((1-?) X
? Y) lt (1-?) f(X) ? f(Y)
for all ? such that 0 lt ? lt 1. f is called
concave ( strictly concave) if f is convex (
strictly convex).
21
Convexity test for function of one variable

A function of one variable f(x) is
convex if
concave if
22
Convexity test for functions of 2 variables
23
Constrained optimization KarushKuhnTucker (KKT)
conditions Consider the problem
maximize z f(X) f(x1, x2,, xn)
subject to g(X) 0 g1(X) 0
g2(X) 0
.
gm(X) 0 (the
non-negativity restrictions, if any, are included
in the above).
24
Let
(where s12, s22,..,sm2 are the non
negative slack variables added to g1(X) 0 ,
g2(X) 0, , gm(X) 0 to make them into
equalities).
We define the Lagrangean function L(X, S, ?)
f(X) ?g(X) S2
f(X) ?1g1(X) s12
?2g2(X)s22) ?mgm(X) sm2
25
KKT necessary conditions for optimality are given
by
26
These are equivalent to the following conditions

(Complementary slackness)
We denote the Lagrangean L(X, ?) by
27
In scalar notation, this is given by
i1,2,.m
?i 0,
I.
II.
III.
IV.
28
The same conditions apply for a minimization
problem also except that now we have
Also in both maximization and minimization
problems, the Lagrange multipliers ?i
corresponding to equality constraints gi(X)
0 must be URS (unrestricted in sign).
29
Sufficiency of the KKT conditions
Required conditions
Sense of optimization
Objective function
Solution space
concave
Convex set
maximization
convex
Convex set
minimization
30
It is simpler to verify whether a function is
concave or convex than to prove that the solution
space is a convex set. We thus give a set of
sufficient conditions that are easier to check
that the solution space is a convex set in terms
of the convexity or concavity of the constraint
functions.
31
Consider the general non-linear problem
Maximize or minimize z f(X)
Subject to
gi(X) 0
i 1,2,.., p
gi(X) ? 0
i p1, p2,.., q
gi(X) 0
i q1, q2,.., r
32
Sufficiency of the KKT conditions
Required conditions
Sense of optimization
f(X) gi(X) ?i
1 i p p1 i q q1 i r

• 0
• 0
• URS

convex concave linear
maximization
concave

• 0
• 0
• URS

convex concave linear
1 i p p1 i q q1 i r
convex
minimization
33
The conditions in the above table represent only
a subset of the conditions given in the earlier
table. The reason is that a solution space may be
convex without satisfying the conditions of the
constraint functions given in the second table.
34
Problem Use the KKT conditions to derive an
optimal solution for the following problem
maximize f(x1, x2) x1 2x2 x23
subject to x1 x2 1
x1 0
x2 0
35
Solution Here there are three constraints
namely, g1(x1,x2) x1x2 - 1
0 g2(x1,x2) - x1 0
g3(x1,x2) - x2 0
Hence the KKT conditions become
?10, ?2 0, ?3 0
36
Note f is concave gi are convex,
maximization problem ? these KKT
conditions are sufficient at the optimum point
?1g1(x1,x2) 0 ?2g2(x1,x2) 0 ?3g3(x1,x2)
0 g1(x1,x2) 0 g2(x1,x2) 0
g3(x1,x2) 0
37
i.e. 1 ?1 ?2 0 (1)
2 3x22 ?1 ?3 0 (2)
?1(x1 x2 1) 0 (3) ?2
x1 0 (4)
?3 x2 0 (5)
x1 x2 1 0 (6)
x1 0 (7)
x2 0 (8)
?1 0 (9) ?2
0 (10) and ?3
0 (11)
38
(1) gives ?1 1 ?2 1 gt0 (using 10) Hence (3)
gives x1 x2 1 (12) Thus both x1, x2 cannot
be zero. So let x1gt0 (4) gives ?2 0. therefore
?1 1 if now x2 0, then (2) gives 2 0 1
?3 0 or ?3 lt 0

not possible Therefore x2 gt 0 hence (5) gives ?3
0 and then (2) gives x22 1/3 so x2 1/v3 And
so x1 1- 1/v3 Max f 1 - 1/v3 2/v3 1/3v3
1 (2 / 3v3 )
39
Maximize f(x) 20x1 10 x2 Subject to x12
x22 1 x1 2x2 2
x1 0, x2 0 KKT conditions become 20 -
2?1 x1 ?2 ?3 0 10 - 2?1 x2 2?2 ?4
0 ?1 (x12 x22 1) 0 ?1 (x1 2x2 2) 0 ?3
x1 0
(0,1)
(4/5,3/5)
40
?4 x2 0 x12 x22 1 x1 2x2
2 x1 0 x2 0
?1 0 ?2 0 ?3
0 ?4 0
41
From the figure it is clear that max f occurs at
(x1, x2) where x1, x2 gt0. ?3 0, ?4 0 suppose
x1 2x2 2 ? 0 ?2 0 ,
therefore we get 20 - 2?1 x10
10 - 2?1
x20 ?1x110 and ?1x25, squaring and adding
we get
?12 125 ?1 5v5 therefore x1
2/v5, x2 1/v5, f 50/v5 gt22
42
?2 ? 0 x1 x2 2 0 Therefore x10,
x21, f 10 Or x1 4/5, x2 3/5,
f22 Therefore max f occurs at x1 2/v5, x2
1/v5
43
Problem Use the KKT conditions to derive an
optimal solution for the following problem
minimize f(x1, x2) x12 x2 subject to
x12 x22 9 x1 x2
1 Solution Here there are two constraints,
namely, g1(x1,x2) x12x22 - 9 0
g2(x1,x2) x1 x2 -1 0
44
Thus the KKT conditions are
as it is a minimization problem

45
Now (from 2) gives
Not possible.
(5)
Hence
and so
Assume . So (1st equation of ) (2)
gives
Since we get x1 0
From (5), we get
2nd equation of (2) says (with
) x2 -3
Thus the optimal solution is The
optimal value is
46
Basic r x1 x2 ?1 ?2 ?1
?2 R1 R2 S1 S2 Sol
22 -8 2 5
-1 -1 0 0 0 0
70
r 1 0 0 0 0
0 0 -1 -1 0 0
0
R1 0 40 -18 1 1 -1
0 1 0 0 0 20
R2 0 -18 10 1 4 0
-1 0 1 0 0 50
S1 0 1 1 0 0 0
0 0 0 1 0
6
S2 0 1 4 0 0 0
0 0 0 0 1
18
r 1 0
x1 0 1
R2 0 0
S1 0 0
S2 0 0
47
Basic r x1 x2 ?1 ?2 ?1
?2 R1 R2 S1 S2 Sol
r 1 0
x1 0
R2 0
x2 0
S2 0
r 1
x1 0
R2 0
x2 0
?2 0
48
Using Excel Solver, the optimum solution is x1
1.604651, x2 4.395349 and the maximum z
230.7209