Newton method and its use in optimization

1
Newton method and its use in optimization

• B.T.Polyak
• Institute for Control Science, Moscow
• boris_at_ipu.rssi.ru
• Ankara, July 2004

2
Outline

• Idea of the method
• History
• Local convergence
• Global behavior and fractals
• Modifications to extend convergence
• Underdetermined systems with applications
• Unconstrained and constrained optimization
• Interior point methods
• Globally convergent version for optimization
• Extensions

3
Idea of the method
4
Example
5
History

• Newton 1669, 1736
• Raphson 1690
• Fourier 1818
• Cauchy 1829, 1847
• Fine 1916
• Bennet 1916
• Ostrowski 1936

6
References

• Ostrowski, Solution of equations and systems of
  equations, 1960
• Ortega, Rheinboldt, Iterative solution of
  nonlinear equations in several variables, 1970
• Kantorovich, Akilov, Functional analysis, 1977
• T.J.Ypma, Historical development of the
  Newton-Raphson method, SIAM Review, 1995, 531-551
• Bibliography for Newtons method
• math.fullerton.edu/mathnews/n2003/newtonsmethod

7
Kantorovichs contribution

• On Newtons method for functional equations,
  Doklady AN SSSR, 1948, 59, No.7, 1237-1240
• Functional analysis and applied mathematics,
  Uspekhi Mat. Nauk, 1948, 3, No. 6, 89-185
• Several other publications, 1949-1960

8
Local convergence
2abc
C
9
Features

• Functional spaces
• Existence theorem
• Quadratic rate of convergence
• Local convergence
• Numerous applications

10
Global behavior
11
Overcoming local nature of the method

• Damped Newton
• Levenberg-Marquardt method (1944)
• Continuous version (Smale, Borkar, Ramm)

12
Underdetermined equations
13
Application convexity principle
14
Examples

• If A has all distinct eigenvalues, then its
  pseudospectrum consists of n convex clouds
• The reachable set of a nonlinear system with
  bounded control is convex
• Optimization problems on a small ball can be
  treated as convex ones

15
Minimization problems
16
Properties

• Local convergence
• Does not distinguish minima, maxima and saddle
  points
• No monotone decrease of f(x)

17
Constrained minimization
18
Selfconcordant functions
19
Nesterov-Nemirovski theorem

• If f(x) is convex, selfconcordant and

20
Interior-point methods
21
Global convergence
22
Convergence results

• For nonconvex problems the method globally
  converges to a local minimum
• Rate of convergence can be estimated for various
  classes of functions
• For all cases the method is more effective than
  the gradient one

23
Various extensions of Newton-Kantorovich method

• Multiple roots
• Data at one point
• Higher-order methods
• Equilibrium problems
• Complementarity problems
• Method in the presence of errors
• Implementation issues
• Complexity

24
Newton basins