Frame, Reproducing Kernel and Learning

1
Frame, Reproducing Kernel and Learning

• Alain Rakotomamonjy
• Stéphane Canu

Perception, Systèmes et Information Insa de
Rouen, 76801 St Etienne du Rouvray France
Alain.Rakoto,Stephane.Canu_at_insa-rouen.fr
http//asi.insa-rouen.fr/arakotom
2
Motivations

• Wavelet-based approximation (wavelet or ridgelet
  networks) are regularization networks?
• Construction of multiresolution scheme of
  approximation
• kernel adapted to the structures of function to
  be learned

3
Motivations Ctd.

• Frame based framework for learning

Approximating highly oscillating structure
Without losing regularity in smooth region
4
Road Map

• Introduction on Frame
• From Frame to Kernels
• From Frame kernels to learning
• Conclusions and perspectives

5
Frame A definition

• H Hilbert Space dot product

A sequence of elements of H
is a frame of H if there exists A,B gt O s.t
A,B are the frame bounds
6
Frame definition Ctd.

• Frame intepretation
• Frame allows stable representation
• as for all f in H

Frame "Basis" linear dependency redundancy
being a dual frame of Fn in H
7
Particular cases of Frame

• Tight Frame
• Frame with bounds s.t AB
• Orthonormal Basis
• AB1
• Riesz Basis
• Frame elements are linearly independent

8
Examples of Frame

• Tight Frame of IR2
• Frame of L2(IR)

F2
F1
F3
Y is an admissible wavelet
9
Road Map

• Introduction on Frame
• From Frame to Kernels
• From Frame kernels to learning
• Conclusions and perspectives

10
Frameable RKHS

• Condition for having a RKHS

Suppose H is a Hilbert space of function
and a frame of H
H is a RKHS if
On a frameable Hilbert Space, this is equivalent
to
The Reproducing Kernel is
11
Construction of Frameable RKHS

• A Practical way to build a RKHS
• F is a Hilbert Space of function

A finite set of F elements such that
?
?
is a RKHS with Fn as frame elements
12
Example of Frameable RKHS

• frameable RKHS included in L2(IR)

Fi L2 function (e.g Fi is a wavelet) span
Fii1N is a RKHS
Example
3 wavelets at same scale j
span a RKHS with kernel
13
Road Map

• Introduction on Frame
• From Frame to Kernels
• From Frame kernels to learning
• Conclusions and perspectives

14
Semiparametric Estimation

• Context

Learning from training set (xi,yi)i1..N
Semiparametric framework
One looks for the minimizer of the risk functional
in a space H spanYii1m H being a RKHS
Under general conditions,
spanYii1m parametric hypothesis space
15
Semiparametric Estimation

• Parametric hyp. space is a frameable RKHS

P is a frameable RKHS spanned by Fn, with P ?
H, H RHKS
Semiparametric estimation on H with P as a
parametric hyp. space
One looks for the minimizer in H of
As spaces are orthogonal, backfitting is
sufficient for estimating f
16
Semiparametric Estimation

• Frame view point
• H frameable
• H defined by kernel K

H P N
P Frameable RKHS, N Frameable RKHS
H
N "unknown component" to be regularized
P ? N due to linear dependency of frame
P "known component" not to regularized
KNKH-KP
P Frameable RKHS
17
Multiscale approximation

• H a frameable RKHS

H is splitted in different spaces Fii1m-1 and
H0
And any space Hi or Fi is a RKHS
Hi Trend Spaces
Fi Details Spaces
18
Multiscale Approximation Ctd.
At each step j, trend obtained at step j-1 is
decomposed in trend and details
H
H2
F2
H1
F1
H0
F0
19
Multiscale Approximation Ctd.

• Validity
• At each step, representer Theorem Hypothesis must
  be verified
• Solution

20
Illustration on toy problem
Function to be learned
Data
xi N points from the random sampling of 0, 10

Algorithm
- SVM Regression
- Multiscale Regularisation networks on Frameable
RKHS
Sin/Sinc based kernel
Wavelet based kernel
21
Results

• N902
• Results are averagerad over 300 experiments and
  normalized with regards to SVM performance

Wavelet Kernel
Sinc Kernel
SVM
1 0.096
0.9297 0.312
0.5115 0.098
L2 error
0.7252 8.022
0.8280 0.025
1 0.028
22
Plots of typical results
23
Road Map

• Introduction on Frame
• From Frame to Kernels
• From Frame kernels to learning
• Conclusions and perspectives

24
Summary

• new design of kernel based on frame elements
• algorithm for multiscale learning
• But
• no explicit definition of kernel
• Time-consuming

25
Future work

• Multidimensional extension
• Tight Frame of multidimensional wavelet
• Using a priori knowledge on the learning problem
• How to choose the frame elements?
• Theoretical justification and analysis of
  multiscale approximation

26
(No Transcript)