Loading...

PPT – 15'053 Thursday, April 25 PowerPoint presentation | free to download - id: 155f22-MjFiZ

The Adobe Flash plugin is needed to view this content

15.053 Thursday, April 25

- Nonlinear Programming Theory
- Separable programming

Handouts Lecture Notes

Difficulties of NLP Models

Linear Program

Nonlinear Programs

Graphical Analysis of Non-linear programs in two

dimensions An example

- Minimize

- subject to

(x - 8)2 (y - 9)2 49 x 2

x 13 x y 24

Where is the optimal solution?

Note the optimal solution is not at a corner

point. It is where the isocontour first hits the

feasible region.

Another example

Minimize (x-8)2 (y-8)2

Then the global unconstrained minimum is

also feasible.

The optimal solution is not on the boundary

of the feasible region.

Local vs. Global Optima

Defn Let x be a feasible solution, then

x is a global max if f(x) f(y) for every

feasible y.

x is a local max if f(x) f(y) for every

feasible y sufficiently close to x (i.e., xj-e

yj xj e for all j and some small e).

There may be several locally optimal solutions.

When is a locally optimal solution also globally

optimal?

- We are minimizing. The objective
- function is convex. The feasible
- region is convex.

Convexity and Extreme Points

We say that a set S is convex, if for every two

points x and y in S, and for every real number ?

in 0,1, ?x (1-?)y e S.

The feasible region of a linear program is

convex.

We say that an element w e S is an extreme point

(vertex, corner point), if w is not the midpoint

of any line segment contained in S.

- Recognizing convex feasible regions
- If all constraints are linear, then the
- feasible region is convex
- The intersection of convex regions is
- convex
- If for all feasible x and y, the
- midpoint of x and y is feasible, then
- the region is convex (except in
- totally non-realistic examples. )

Which are convex?

Convex Functions Convex Functions f(l y (1-

l)z) l f(y) (1- l)f(z) for every y and z and

for 0 l 1. e.g., f((yz)/2) f(y)/2

f(z)/2 We say strict convexity if sign is lt

for 0lt ? lt1.

Line joining any points is above the curve

Convex Functions Convex Functions f(l y (1-

l)z) l f(y) (1- l)f(z) for every y and z and

for 0 l 1. e.g., f((yz)/2) f(y)/2

f(z)/2 We say strict convexity if sign is lt

for 0lt ? lt1.

Line joining any points is above the curve

Classify as convex or concave or both or neither.

What functions are convex?

- f(x) 4x 7 all linear functions
- f(x) 4x2 13 some quadratic

functions - f(x) ex
- f(x) 1/x for x gt 0
- f(x) x
- f(x) - ln(x) for x gt 0
- Sufficient condition f(x) gt 0 for all x.

What functions are convex?

- If f(x) is convex, and g(x) is convex.
- Then so is
- h(x) a f(x) b g(x) for agt0, bgt0.

- If y f(x) is convex, then
- (x,y) f(x) y is a convex set

Local Maximum (Minimum) Property

- A local max of a concave function on a convex
- feasible region is also a global max.
- A local min of a convex function on a convex

feasible - region is also a global min.
- Strict convexity or concavity implies that the

global - optimum is unique.
- Given this, we can exactly solve
- Maximization Problems with a concave

objective - function and linear constraints
- Minimization Problems with a convex

objective - function and linear constraints

Which are convex feasible regions?

(x, y) y x2 2 (x, y) y x2 2 (x,y) y

x2 2

More on local optimality

- The techniques for non-linear optimization
- minimization usually find local optima.
- This is useful when a locally optimal
- solution is a globally optimal solution
- It is not so useful in many situations.
- Conclusion if you solve an NLP, try to
- find out how good the local optimal
- solutions are.

Solving NLPs by Excel Solver

Finding a local optimal for a single variable NLP

Solving NLP's with One Variable

max f(?) s.t. a ?

b Optimal solution is either

a boundary point or satisfies

f'(?) 0 and f ?(?) lt 0.

Solving Single Variable NLP (contd.)

If f(?) is concave (or simply unimodal) and

differentiable

max f(?) s.t. a ? b

Bisection (or Bolzano) Search

- Step 1. Begin with the region of uncertainty

for ? as a, - b. Evaluate f ' (?) at the midpoint ?? (ab)/2.

- Step 2. If f ' (??) gt 0, then eliminate the

interval up to ??. - If f'(??) lt 0, then eliminate the interval beyond

??.

- Step 3. Evaluate f'(?) at the midpoint of the

new - interval. Return to Step 2 until the interval of

uncertainty - is sufficiently small.

- Unimodal Functions
- A single variable function f is
- unimodal if there is at most one local
- maximum (or at most one local

minimum) .

Other Search Techniques

- Instead of taking derivatives (which may be
- computationally intensive), use two function
- evaluations to determine updated interval.
- Fibonacci Search
- Step 1. Begin with the region of uncertainty

for ? as a, - b. Evaluate f (?1) and f (?2) for 2

symmetric points - ?1lt?2.
- Step 2. If f (?1) f (?2), then eliminate the

interval up t?1. - If f (?1) gt f (?2), then eliminate the

interval beyond ?2. - Step 3. Select a second point symmetric to the

point - already in the new interval, rename these

points ?1 and - ?2 such that ?1lt?2 and evaluate f (?1) and f

(?2). Return - to Step 2 until the interval is sufficiently

small.

On Fibonacci search

1, 1, 2, 3, 5, 8, 13, 21, 34 At iteration

1, the length of the search interval is the

kth fibonacci number for some k At iteration

j, the length of the search interval is the

k-j1 fibonacci number. The technique converges

to the optimal when the function is unimodal.

Finding a local maximum using Fibonacci Search.

Length of search

interval

Where the maximum may be

The search finds a local maximum, but not

necessarily a global maximum.

The search finds a local maximum, but not

necessarily a global maximum.

Number of function evaluations in Fibonacci Search

- As new point is chosen symmetrically, the

length lk of - successive search intervals is given by lk

lk1 lk2 . - Solving for these lengths given a final

interval length of - 1, ln 1, gives the Fibonacci numbers 1, 2,

3, 5, 8, 13, 21, - 34,
- Thus, if the initial interval has length 34, it

takes 8 - function calculations to reduce the interval

length to 1. - Remark if the function is convex or unimodal,

then - fibonacci search converges to the global

maximum

Separable Programming

- Separable programs have the form

Max

st

Each variable xj appears separately, one in each

function gij and one in each function fj in the

objective.

Each non-linear function is of a single variable.

Examples of Separable Programs

f(x1,x2) x1(30-x1)x2(35-x2)-2x12-3x22

3

f(x1,x2) x15 -18e-x2-4x2

x1

f(x1,x2,x2) lnx15-sinx2-x3e-x3)7x1-4

Approximating a non-linear function with a

piecewise linear function.

- Aspect 1. Choosing the approximation.
- Aspect 2. When is the piecewise linear
- approximation a linear program in
- disguise?

Approximating a non-linear function of 1 variable

Approximating a non-linear function of 1

variable the ? method

Choose different values of x to approximate the

x-axis

Approximate using piecewise linear segments

More on the ? method

a1 -3, f(a1) -20 a2 -1 f(a2) -7 1/3

Suppose that for 3 x -1, we represent x

has?1 (-3) ?2 (-1) where?1 ?2 1 and?1,?2 0

Then we approximate f(x) as ?1 (-20) ?2 (-7 1/3)

More on the ? method

Suppose that for 1 x 1, we represent x

has?2 (-3) ?3 (-1) where?2 ?3 1 and?2,?3 0

a2 -1, f(a2) -7 1 /3 a3 1 f(a3) -2 2/3

How do we approximate f( ) in this interval?

What if 3 x 1?

Almost the ? method

Original problem min x3/3 2x 5 more

terms s.t. -3 x 3 many more constraints

a1 -3 a2 - 1 a3

1 a4 3 f(a1) -20 f(a2)

-7 1/3 f(a3) -2 2/3 f(a4) 4

Approximated problem min?1f(a1) ?2f(a2)

?3f(a3) ?4f(a4) more

linear terms s.t. ?1 ?2 ?3 ?4 1 ? 0

many more constraints

Why the approximation is incorrect

Approximated problem min?1f(a1) ?2f(a2)

?3f(a3) ?4f(a4) more

linear terms s.t. ?1?2 ?3?4 1 ? 0

Consider?1 1/2 ?20 ?31/2 ?4 0

The method gives the correct approximation

if only two consecutive?s are positive.

Adjacency Condition 1. At most two weights (?s)

are positive 2. If exactly two weights (?s) are

positive, then they are ?j and ?j1 for some

j 3. The same condition applies to every

approximated function.

- Approximating a non-linear objective
- function for a minimization NLP.
- original problem minimize f(y) y ? P

- Suppose that y Sj?jaj ,
- where Sj?j 1

and ? gt 0 . - Approximate f(y).
- minimize Sj?jf(aj) Sj?jaj? P
- Note when given a choice of representing y in
- alternative ways, the LP will choose one that
- leads to the least objective value for the
- approximation.

For minimizing a convex function, the ?-method

automatically satisfies the additional adjacency

property.

min Z ?1f(a1) ?2f(a2) ?3f(a3) ?4f(a4)?5f(a5)

s.t. ?1 ?2 ?3

?4 ?5 1 ? 0

adjacency condition

other constraints

Feasible approximated objective functions without

the adjacency conditions

min Z ?1f(a1) ?2f(a2) ?3f(a3) ?4f(a4)?5f(a5)

s.t. ?1 ?2 ?3

?4 ?5 1 ? 0

other constraints

But a minimum in this case always occurs on the

piecewise linear curve.

min Z ?1f(a1) ?2f(a2) ?3f(a3) ?4f(a4)?5f(a5)

s.t. ?1 ?2 ?3

?4 ?5 1 ? 0

other constraints

Separable Programming (in the case of linear

constraints)

Max f ( )x s.t. Dx d x ³ 0

- Begin with an NLP
- Transform to Separable
- Approximate using the ? Method

n

Max ? fj (xj) s.t. Dx d x ³ 0

j1

- Approximation
- Re-express in terms of ? variables

and the adjacency conditions

If the original problem is concave, then you can

drop the adjacency conditions (they are

automatically satisfied.)

How can one construct separable functions?

Term Substitution Constraints

Restriction

None

None

None

Transformation Examples

Ex (x1 x2 x3)6

Substitute y6 and let y x1 x2 x3

x1x22

let y1

Ex

and let y2

1 x3

and add the constraint

NLP Summary

- Convex and Concave functions as well as convex

sets - are important properties
- Bolzano and Fibonacci search techniques
- used to solve single variable unimodal

functions - Separable Programming
- nonlinear objective and nonlinear

constraints that - are separable
- General approximation technique