Natural Gradient Approach to Optimization and its use in Blind source separation

1
Natural Gradient  Approach to Optimization and
its use in Blind source separation

• BY Maha Elsabrouty
• Supervisors Prof. Dr. Tyseer Aboulnasr
• Prof. Dr Martin Bouchard

2
Motivation

• Optimization is the central idea behind many
  problems in science and engineering.
• Successful Optimization techniques exploit the
  given structure of the underlying space.

3
Motivation

• Manifolds endowed with a metric structure, i.e.
  Riemannian, arise naturally in applications in
  signal processing, mechanics and control theory.
• In such constrained problems, the super-linear
  convergence speed of the classical conjugate
  gradient and Newton algorithms is lost as the
  structure of the surface is ignored.

4
Riemannian manifold Definitions and Properties

• What is a manifold?

A differentiable space that is locally Euclidean

• What is a Riemannian manifold?

A manifold that posses a metric tensor g that
is both symmetric and positive definite. The
metric tensor is used to calculate the actual
distance on the Riemannian surface.
5

• The rest of the Riemannian geometry stems from
  its main characteristic, i.e. Curvature.
• Basically curvature vector indicates the
  direction in which a certain curve under
  inspection is turning.

6

• At each point p on the manifold we can find a
  tangent vector in each direction. The set of such
  vectors constitute tangent bundle Tp . This
  Euclidean space contains the first order
  differentiation of any curve passing by the point
  p.

Figure 1
7

• Because of this curvature a lot of laws governing
  motion on the Euclidean space are no longer valid
  or at least more complicated. How?? !!!
• In Euclidean spaces, tangents are naturally
  isometric. There is a natural connection between
  the tangent bundles. Moving one tangent vector
  from a point to another does not require more
  than the original point and end point of the
  displacement.

8

• Due to curvature such connection has to be
  imposed.
• Moving the tangent along a curve on the
  Riemannian manifold involves displacing it from
  its base to the new point and then subtracting
  the normal component.

9
Figure 2
The operation of creating the new tangent vector
at the new point is referred to as
Parallel transposition
10

• For every Riemannian manifold there exists a
  unique connection named Levi-Civitae
  connection. This curvature is symmetric and
  torsion free.
• This connection is intrinsic and defined by the
  curvature coefficients known as Christoffel
  coefficients that are in their turn
  defined by the metric coefficients g.

11

• In terms of Christoffel coefficients , we can
  define the Levi-Civitae connection using the
  canonical basis of the patch
  chart defined on the manifold
  as

The significance of such operation is the use of
the whole set of Christoffel coefficients
carrying within all the normal information about
the curvature of the manifold and thus enabling
the movement on the tangential space .
12

• The importance of this connection lies in the
  fact that it defines a path ( a unique curve)
  along the Riemannian space that has two powerful
  properties
• Has minimum curvature length between two
• Parallel transposition between two points lying
  on this curve is carried along this curvature.
• Geodesics are curves defined by the equation

13

• The actual distance between two points on the
  Riemannian manifold is calculated along
  geodesics. They are the curves of shortest
  distance possible connection the two points.
• Geodesics are actually the curves along which
  optimization is carried. Search directions for
  the Riemannian manifold has to be carried along
  geodesics.

14
Examples of geodesics
Figure 3
Figure 4
Figure 5
15
Goedesics Highlight of Characteristics

• Geodesics have no tangential curvature.
• Geodesics curvatures depends entirely on the
  surface of the manifold (intrinsic).
• Between any two points on the manifold there
  exists a geodesic of shortest length that is both
  simple and convex.

16

• 4. Geodesics are homogeneous, every sub-segment
  on a geodesic carries all the geodesic
  characteristics.
• 5. Geodesics transposes its own tangent vector to
  itself.
• 6. Moving along a geodesic from one point to
  another is carried out by performing exponential
  mapping.

17
Optimization On Riemannian manifolds

• The main steps involving first order optimization
  along Riemannian manifold includes
• Calculating the actual gradient on the Riemannian
  manifold according to the equation
• Moving along the appropriate search direction,
  i.e. along the geodesic by exponential mapping.

18
The Actual value of natural gradient

• The regular stochastic gradient method speed of
  convergence is affected by the initial points and
  the step size used. The slop of the cost function
  varies for small changes in the parameter.
• Natural gradient modifies the slope (takes into
  account the actual structure of the optimization
  space) and thus provide a better approximation to
  the steepest descent direction

19
Figure 6
20
Natural Gradient Adaptation

• One of main application of natural gradient is
  BSS.
• In Such a problem we need to calculate the best
  value for the de-mixing matrix W which is both
  real and invertible , the inverse of W is the
  mixing matrix H.

Figure 7
21

• The natural gradient is a first order gradient
  calculated along the steepest descent direction
  of the Riemannian manifold and thus follows the
  equation
• The main task to calculate the metric g.
• In order to do this we have to use the algebraic
  form of the Riemannian manifold, namely Lie
  groups.

22

• The Lie group right-invariance implies that the
  metric is the same on all the points of the
  manifold

• The identity I on the manifold is the
  0-diffeomorphic point at which the Riemannian
  manifold holds the normal co-ordinates and is
  equal to the Euclidean. i.e. the metric tensor G
  I.
• let dW be a small deviation of the demixing
  matrix from W to W dW then the tangent space TW
  (the Lie algebra of GL(n)) contains all such
  small deviations dWij .

23

• let gW (dW,dW) to be the metric at the point W

• Now if we map the matrix W to the identity matrix
  I, we have to right multiply W by then
  the tangent dX dW at the identity
  correspond to dW at W under the right-invariance
  of Lie-groups and has the same length.

24

• As dX is calculated at the identity point. Its
  metric can be simply calculated at the Euclidean
  space

• Thus the metric at point W is

• Then we can extract the metric from the
  above equation

25

• What we actually need to calculate the derivative
  on the Riemannian manifold is the inverse of g
  which is given by