A Framework For Pruning Deep Neural Networks Using Energy-Based Models

2021-02-25 · Hojjat Salehinejad, Shahrokh Valaee

A typical deep neural network (DNN) has a large number of trainable parameters. Choosing a network with proper capacity is challenging and generally a larger network with excessive capacity is trained. Pruning is an established approach to reducing the number of parameters in a DNN. In this paper, we propose a framework for pruning DNNs based on a population-based global optimization method. This framework can use any pruning objective function. As a case study, we propose a simple but efficient objective function based on the concept of energy-based models. Our experiments on ResNets, AlexNet, and SqueezeNet for the CIFAR-10 and CIFAR-100 datasets show a pruning rate of more than $50\%$ of the trainable parameters with approximately $<5\%$ and $<1\%$ drop of Top-1 and Top-5 classification accuracy, respectively.

📄 PDF Abstract BibTeX arXiv:2102.13188

Code (0)

등록된 구현이 없습니다.

Tasks

global-optimization

Methods 이 논문이 사용한 방법론

Pruning 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Average Pooling 설명 없음

Fire Module 설명 없음

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Residual Connection 설명 없음

Xavier Initialization Xavier Initialization, or Glorot Initialization, is an initialization scheme for neural networks. Biases are initialized be 0 and the weights $W\_{ij}$ at each layer are…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks

2020-06-07 · Hojjat Salehinejad, Shahrokh Valaee

Dropout is a well-known regularization method by sampling a sub-network from a larger deep neural network and training different sub-networks on different subsets of the data. Inspired by the dropout concept, we propose …

R-TOSS: A Framework for Real-Time Object Detection using Semi-Structured Pruning

2023-03-03 · Abhishek Balasubramaniam, Febin P Sunny, Sudeep Pasricha

Object detectors used in autonomous vehicles can have high memory and computational overheads. In this paper, we introduce a novel semi-structured pruning framework called R-TOSS that overcomes the shortcomings of state-…

Autonomous Vehicles Object object-detection Object Detection+1

A Theoretical Framework for Energy-Aware Gradient Pruning in Federated Learning

2026-03-23 · Emmanouil M. Athanasakos arxiv

Federated Learning (FL) is constrained by the communication and energy limitations of decentralized edge devices. While gradient sparsification via Top-K magnitude pruning effectively reduces the communication payload, i…

Federated Learning

Pruning of Convolutional Neural Networks Using Ising Energy Model

2021-02-10 · Hojjat Salehinejad, Shahrokh Valaee

Pruning is one of the major methods to compress deep neural networks. In this paper, we propose an Ising energy model within an optimization framework for pruning convolutional kernels and hidden units. This model is des…

Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantization

2023-12-23 · Konstantinos Balaskas, Andreas Karatzas, Christos Sad, Kostas Siozios 외

Deep Neural Networks (DNNs) have shown significant advantages in a wide variety of domains. However, DNNs are becoming computationally intensive and energy hungry at an exponential pace, while at the same time, there is …

Quantization Reinforcement Learning (RL)