Convolutional%20Deep%20Belief%20Networks%20for%20Scalable%20Unsupervised%20Learning%20of%20Hierarchical%20Representations%20%20Honglak%20Lee,%20Roger%20Grosse,%20Rajesh%20Ranganath,%20and%20Andrew%20Y.%20Ng%20ICML%202009

1
Convolutional Deep Belief Networks for Scalable
Unsupervised Learning of Hierarchical
Representations Honglak Lee, Roger Grosse,
Rajesh Ranganath, and Andrew Y. Ng ICML 2009

• Presented by Mingyuan Zhou
• Duke University, ECE
• September 18, 2009

2
Outline

• Motivations
• Contributions
• Backgrounds
• Algorithms
• Experiment results
• Deep Vs Shallow
• Conclusions

3
Motivations

• To Learn hierarchical models which simultaneously
  represent multiple levels, e.g., pixel
  intensities, edges, object parts, objects, and
  beyond can be represented by layers from low to
  high.
• Combining top-down and bottom-up processing of an
  image.
• Limitations of deep belief networks (DBNs)
• Scaling DBNs to realistic-size images remains
  challenging images are high-dimentional and
  objects can appear at arbitrary locations in
  images.

4
Contributions

• Convolutional RBM feature detectors are shared
  among all locations in an image.
• Probabilistic max-pooling in a probabilistic
  sound way allowing higher-layer units to cover
  larger areas of the input.
• The first translation invariant hierarchical
  generative model supporting both top-down and
  bottom-up probabilistic inference and sales to
  realistic image sizes.

5
Backgrounds Restricted Boltzmann Machine (RBM)
(binary v)
(real-value v)

• Giving the visible layer, the hidden units are
  conditionally independent, and vise versa.
• Efficient block Gibbs sampling can be performed
  by alternately sampling each layers units.
• Computing the exact gradient of the
  log-likelihood is intractable, so the contrastive
  divergence approximation is commonly used.

6
Backgrounds Deep belief network (DBN)

• In a DBN, two adjacent layers have a full set of
  connections between them, but no two units in the
  same layer are connected.
• A DBN can be formed by stacking RBMs.
• An efficient algorithm for training DBNs (Hinton
  et al., 2006) greedily training each layer,
  from lowest to highest, as an RBM using the
  previous layer's activations as inputs.

7
Algorithms Convolutional RBM (CRBM)
8
Algorithms Probabilistic max-pooling
9
Algorithms Probabilistic max-pooling

• Each unit in a pooling layer computes the maximum
  activation of the units in a small region of the
  detection layer.
• Shrinking the representation with max-pooling
  allows higher-layer representations to be
  invariant to small translations of the input and
  reduces the computational burden.
• Max-pooling was intended only for feed-forward
  architectures. A generative model of images which
  supports both top-down and bottom-up inference is
  of interest.

10
Algorithms Sparsity regulations

• Only a tiny fraction of the units should be
  active in relation to a given stimulus.
• Regularizing the objective function to encourage
  each of the hidden units to have a mean
  activation close to some small constant .

11
Algorithms Convolutional DBN (CDBN)

• CDBN consists of several max-pooling-CRBMs
  stacked on top of one another.
• Once a given layer is trained, its weights are
  frozen, and its activations are used as input to
  the next layer.

12
Hierarchical probabilistic inference
13
Experimental Results natural images
14
Experimental Results image classification
15
Experimental Results unsupervised learning of
object parts
16
Experimental Results Hierarchical probabilistic
inference
17
Deep Vs Shallow
.

• From Jason Westons slides DEEP LEARNING VIA
  SEMI-SUPERVISED EMBEDDING, ICML 2009 WORKSHOP ON
  LEARNING FEATURE HIERARCHIES

From Francis Bachs slides Convex sparse
methods for feature hierarchies, ICML 2009
WORKSHOP ON LEARNING FEATURE HIERARCHIES
18
Conclusions

• Convolutional deep belief network
• A scalable generative model for learning
  hierarchical representations from unlabeled
  images.
• Performing well in a variety of visual
  recognition tasks.