A Systematic DNN Weight Pruning Framework
using Alternating Direction Method of
Multipliers
Abstract. Weight pruning methods for deep neural networks (DNNs)
have been investigated recently, but prior work in this area is mainly
heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we
present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the
weight pruning problem of DNNs as a nonconvex optimization problem
with combinatorial constraints specifying the sparsity requirements, and
then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed
into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be
solved analytically. Besides, our method achieves a fast convergence rate.
The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we
achieve 71.2× weight reduction without accuracy loss. On the AlexNet
model for the ImageNet data set, we achieve 21× weight reduction without accuracy loss. When we focus on the convolutional layer pruning for
computation reductions, we can reduce the total computation by five
times compared with the prior work (achieving a total of 13.4× weight
reduction in convolutional layers). Our models and codes are released at
https://github.com/KaiqiZhang/admm-pruning