Geometric Methods for Learning and Memory

1
Geometric Methods for Learning and Memory

• A thesis presented
• by
• Dimitri Nowicki
• To Universite Paul Sabatier
• in partial fulfillment
• for the degree of Doctor es Science
• in the subject of
• Applied Mathematics

2
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

3
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

4
Models and Algorithms requiring Geometric Approach

• Kalmanlike filters
• Blind Signal Separation
• Feed-Forward Neural Networks
• Independent Component Analysis

5
Introduction
Spaces emerging in learning problems

• Riemannian spaces
• Lie groups and homogeneous spaces
• Metric spaces without any Riemannian structure

6
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

7
Outline

• Some facts from Riemannian geometry
• Optimization algorithms
• Smooth
• Nonsmooth
• Implementation
• The case of Submanifolds
• Computing exponential maps
• Computing Hessian etc.

8
Some concepts from Riemannian Geometry

• Geodesics

9
Exponential map
10
Parallel transport

• Computing parallel transport using an exponential
  map

Where u such that
11
Newton Method for Geometric optimization
The modified Newton operator
12
Wolfe condition for Riemannian manifolds
13
Global convergence of modified Newton method
14
Nonsmooth methods

• The subgradient

15
The r-algorithm
.
Here
16
Problem of constrained optimization

• Equality constraints

17
Classical (extrinsic) methods

• The Lagrangian

Newton-Lagrange method
Sequential quadratic programming
18
Classical methods

• Penalty functions and the augmented Lagrangian

19
Advantages of Geometric methods

• Dimension of the manifold is n-m against nm in
  the case of Lagrangian-based methods
• We may have convex function in the manifold even
  if the Lagrangian is non-convex
• Geometric Hessian may be positive-definite even
  if the classical one is not

20
Implementation The case of Submanifolds
21
Hamilton Equations for the Geodesics

• The Lagrangian

The Hamiltonian
22
Hamilton Equations for the Geodesics
23
Lagrange equation are also constrained Hamiltonian

• We can rewrite Lagrange equations in the form

24
Symplectic Numerical Integration

• A transformation is called symplectic if it
  preserves following differential 2-form

25
Implicit Runge-Kutta Integrators
y(x,p)
The IRK method is called symplectic if associated
transformation preserves
26
The Gauss method of order 4
i1 i2
j1 1/4
j2 1/4
1/2 1/2
27
Backward error analysis
28
Covariant Derivative on the Submanifold
29
Computing the constrained Hessian

• Direct computation

where
Mixed computation
30
Example of geometric iterations
31
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

32
Neural Associative memory

• Hopfield-type auto-associative memory. Memorized
  vectors are bipolar vk?-1, 1 n, k1m. Suppose
  these vectors are columns of n?m matrix V. Then
  synaptic matrix C of the memory is given by

Associative recall is performed using following
procedure the input vector x0 is a starting
point of the iterations
where f is a monotonic odd function such that
33
Attraction radius

• We will call the stable fixed point of this
  discrete-time dynamical system an attractor. The
  maximum Hamming distance between x0 and a
  memorized pattern vk such that the examination
  procedure still converges to vk is called an
  attraction radius.

34
Problem statement
35
Generalized averaging on the manifold
argmin
argmin
36
Computing generalized average on the Grassmann
manifold
Generalized averaging as an optimization problem
Transforming objective function
37
Statistical estimation
38
Statistical estimation
39
Experimental results the simulated data
Nature of the data

• n256 for all experiments

40
Experimental results simulated data
41
Experimental results simulated data
Frequencies of attractors of associative
clustering network for different m, p8
42
Experimental results simulated data
Frequencies of attractors of associative
clustering network for different p, and mp
43
Experimental results simulated data

• Distinction coefficients of attractors of
  associative clustering network for different p,
  and mp

44
The MNIST database data description

• Gray-scale images 28?28
• 10 classes digits from 0 to 9
• Training sample 60000 images
• Test sample10000 images
• Before entering to the network images were
  tresholded to obtain 784-dimensional bipolar
  vectors

45
Experimental results the MNIST database

• Example of handwritten digits from MNIST database

46
Experimental results the MNIST database

• Generalized images of digits found by the network

47
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

48
Kernel AM

• The main algorithm

49
Kernel AM

• The Basic Algorithm (Continued)

50
Algorithm Scheme
51
Experimental Results

• Gaussian Kernel

52
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM Quotient spaces for
  Signal Processing
• Application Electronic Nose

53
Model of Signal
54
Signal Trajectories in the phase space
55
The Manifold
56
(No Transcript)
57
Example of Signal Processing
58
Outline

• Introduction
• Geodesics, Newton Method and Geometric
  Optimization
• Generalized averaging over RM and Associative
  memories
• Kernel Machines and AM
• Quotient spaces for Signal Processing
• Application Electronic Nose

59
Application for Real-Life Problem
Electronic Nose QCM Setup overview
Variance Distribution between principal Components
60
Chemical images in space spanned by first 3 PCs