0. Paper List
This list is designed for beginners who are interested in learning more about the technical details behind Torch-Pruning & DepGraph. DepGraph is actually not a pruning algorithm itself. Rather, it serves as a grouping algorithm used to analyze the dependency in networks. To enable pruning, DepGraph should be combined with existing pruning methods like Magnitude Pruning or Taylor Pruning. Most of the ideas in these papers have been implemented in Torch-Pruning.
For a more comprehensive paper list, please refer to MingSun-Tse/Efficient-Deep-Learning.
| Title & Authors | Figure | Implementation in Torch-Pruning |
|---|---|---|
 DepGraph: Towards Any Structural Pruning Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang National University of Singapore "We propose a general and fully automatic method, Dependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning." |
 |
DependencyGraph GroupNormImportance GroupTaylorImportance GroupHessianImportance |
 Optimal Brain Damage Yann LeCun, John Denker, Sara Solla AT&T Bell Laboratories "The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error." |
 |
HessianImportance |
 Pruning Convolutional Neural Networks for Resource Efficient Inference Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz NVIDIA "We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with finetuning by backpropagation—a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters." |
 |
TaylorImportance Iterative Pruning |
|
Pruning Filters for Efficient ConvNets Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf University of Maryland "We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications." |
 |
MagnitudeImportance |
 Learning Efficient Convolutional Networks Through Network Slimming Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang Tsinghua University "In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way." |
 |
BNScaleImportance |
 Neural Pruning via Growing Regularization Huan Wang, Can Qin, Yulun Zhang, Yun Fu Northeastern University "Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring." |
 |
GrowingRegPruner |