Phase transitions in vector quantization

1
Phase transitions in vector quantization
Anarta Ghosh Nordic Bioscience Denmark
Aree Witoelar, Michael Biehl University of
Groningen Netherlands
2
Outline

• Vector Quantization
• Model and Analysis
• Phase transitions
• Extensions to Neural Gas
• Conclusions

3
Vector Quantization

• Representation of P data with K prototypes

Minimize cost function H(W)
data
Example Winner Takes All
Euclidean distance to nearest prototype
Data set
Prototypes
4
Model
Two Gaussian clusters of high dimensional data N
Random vectors ? ? RN according to
center vectors B1, B2 ? RN
separation l
prior prob. p1, p2 p1 p2 1
variance v1, v2
Separable in projection to (B1 , B2) plane
5
Offline/batch learning

• Offline training use all data in the training
  set for each update step
• To avoid being trapped in local minima, thermal
  noise is introduced, controlled by formal
  temperature T.
• Example the Langevin dynamics

white noise
Here we study typical behavior of a stochastic
process on H(W) Stochastic processes, including
the above dynamics, reach a well-defined thermal
equilibrium (stationary state at t ? 8 )
6
Statistical physics analysis

• Thermal equilibrium configuration W is observed
  with probability
• T controls the minimization of H(W)
• High T broad distribution of W
• Low T sharply peaked around minima of H(W)
• Thermal average for one data set D can be derived
  from ln Z

inverse temperature
The normalization Z is called the partition
function
volume element
Volume element
7
Statistical physics analysis

• Perform an average over all possible data sets
• We can get exact results for the high
  temperature limit

rescaled number of examples
8
Results
R11
Q11
Order parameters
Q22
R21
Q12
Training set size

• System of two prototypes
• p1 0.8, p2 0.2, l 1

H(W)
9
Phase transition
unspecialized
specialized

• Below critical training set size , any
  optimization strategies
• based on H(W) will fail to detect cluster
  structure

10
More phase transitions

• System with 3 prototypes First order phase
  transition

R11
R21
R31
11
Neural Gas

• Extension to Neural Gas

Annealing schemes do not require
to detect structure at small ?
12
Conclusions

• Exact analysis of off-line VQ learning in a model
  situation in the high temperature limit
• Below a critical training set size, no learning
  is possible
• Existence of phase transitions
• 2 prototypes Continuous phase transition
• gt3 prototypes Discontinuous phase transition,
  metastable states
• Neural Gas annealing schemes are promising
  strategies for practical optimization
• Next off-line learning at low temperature

13
End