Brain Damage: Algorithms for Network Pruning

1
Brain Damage Algorithms for Network Pruning

• Andrew Yip
• HMC Fall 2003

2
The Idea

• Networks with excessive weights over-train on
  data. As a result, they have poor generalization.
• Create a technique that can effectively reduce
  the size of the network without reducing
  validation.
• Hopefully, by reducing the complexity, network
  pruning can increase the generalization
  capabilities of the net.

3
History

• Removing weights means to set them to 0 and
  freeze them
• First attempt at network pruning removed weights
  of least magnitude
• Minimize cost function composed of both the
  training error and the measure of network
  complexity

4
Lecuns Take

• Derive a more theoretically sound technique for
  weight removal order using the derivative of the
  error function

5
Computing the 2nd Derivatives

• Network expressed as
• Diagonals of Hessian
• Second Derivatives

6
The Recipe

• Train the network until local minimum is obtained
• Compute the second derivatives for each parameter
• Compute the saliencies
• Delete the low-saliency parameters
• Iterate

7
Results
Results of OBD Compared to Magnitude-Based Damage
8
Results Continued
Comparison of MSE with Retraining versus w/o
Retraining
9
Lecons Conclusions

• Optimal Brain Damage results in a decrease in the
  number of parameters by up to four general
  recognition accuracy increased.
• OBD can be used either as an automatic pruning
  tool or an interactive one.

10
Babak Hassibi Return of Lecun

• Several problems arise from Lecuns simplifying
  assumptions
• For smaller sized networks, OBD chooses the
  incorrect parameter to delete
• It is possible to recursively calculate the
  Hessian, yielding a more accurate approximation.

11
Insert Math Here
(I have no idea what Im talking about)
12
The MONKs Problems

• Set of problems involving classifying artificial
  robots based on six discrete valued attributes
• Binary Decision Problems (head_shape
  body_shape)
• Study performed in 1991 Back-propagation with
  weight decay found to be most accurate solution
  at the time.

13
Results Hassibi Wins
14
References

• Le Cun, Yann. Optimal Brain Damage. ATT Bell
  Laboratories, 1990.
• Hassibi, Babak, Stork, David. Optimal Brain
  Surgeon and General Network Pruning. Ricoh
  California Research Center. 1993.
• Thrun, S.B. The MONKs Problems. CMU. 1991.

15
Questions?
(Brain Background Courtesy Brainburst.com)