Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
To solve ever more complex problems, Deep Neural Networks are scaled to billions of parameters, leading to huge computational costs. An effective approach to reduce computational requirements and increase efficiency is to prune unnecessary components of these often over-parameterized networks. Previous work has shown that attribution methods from the field of eXplainable AI serve as effective means to extract and prune the least relevant network components in a few-shot fashion. We extend the current state by proposing to explicitly optimize hyperparameters of attribution methods for the task of pruning, and further include transformer-based networks in our analysis. Our approach yields higher model compression rates of large transformer- and convolutional architectures (VGG, ResNet, ViT) compared to previous works, while still attaining high performance on ImageNet classification tasks. Here, our experiments indicate that transformers have a higher degree of over-parameterization compared to convolutional neural networks. Code is available at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch.
Code (1)
Tasks
Model CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agno…
Decision MakingFeature ImportanceFraud DetectionSequential Decision Making+2Explaining the Reasoning of Large Language Models Using Attribution Graphs
Large language models (LLMs) exhibit remarkable capabilities, yet their reasoning remains opaque, raising safety and trust concerns. Attribution methods, which assign credit to input features, have proven effective for e…
Decision MakingMolLedger: An Additive Graph Neural Network with Chemically Grounded ADME Attributions
Optimizing absorption, distribution, metabolism, and excretion (ADME) is an important part of small molecule drug discovery. Many machine learning models have been built to predict ADME properties to facilitate this opti…
Graph Neural NetworkDrug DiscoveryOn Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
BERT, as one of the pretrianed language models, attracts the most attention in recent years for creating new benchmarks across GLUE tasks via fine-tuning. One pressing issue is to open up the blackbox and explain the dec…
General ClassificationSentiment AnalysisA Granger-Causal Perspective on Gradient Descent with Application to Pruning
Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This arti…