Adaptive Dropout for Pruning Conformers

2024-12-06 · Yotaro Kubo, Xingyu Cai, Michiel Bacchiani

This paper proposes a method to effectively perform joint training-and-pruning based on adaptive dropout layers with unit-wise retention probabilities. The proposed method is based on the estimation of a unit-wise retention probability in a dropout layer. A unit that is estimated to have a small retention probability can be considered to be prunable. The retention probability of the unit is estimated using back-propagation and the Gumbel-Softmax technique. This pruning method is applied at several application points in Conformers such that the effective number of parameters can be significantly reduced. Specifically, adaptive dropout layers are introduced in three locations in each Conformer block: (a) the hidden layer of the feed-forward-net component, (b) the query vectors and the value vectors of the self-attention component, and (c) the input vectors of the LConv component. The proposed method is evaluated by conducting a speech recognition experiment on the LibriSpeech task. It was shown that this approach could simultaneously achieve a parameter reduction and accuracy improvement. The word error rates improved by approx 1% while reducing the number of parameters by 54%.

📄 PDF Abstract BibTeX arXiv:2412.04836

Code (0)

등록된 구현이 없습니다.

Tasks

speech-recognition Speech Recognition

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Pruning 설명 없음

Adaptive Dropout Adaptive Dropout is a regularization technique that extends dropout by allowing the dropout probability to be different for different units. The intuition is that there may be…

EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks

2020-06-07 · Hojjat Salehinejad, Shahrokh Valaee

Dropout is a well-known regularization method by sampling a sub-network from a larger deep neural network and training different sub-networks on different subsets of the data. Inspired by the dropout concept, we propose …

Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization

2020-10-11 · Jiyang Xie, Zhanyu Ma, and Jianjun Lei, Guoqiang Zhang 외

Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance o…

Network Pruning text-classification Text Classification

Targeted Dropout

2018-10-20 · NIPS Workshop CDNNRIA 2018 · Aidan N. Gomez, Ivan Zhang, Kevin Swersky, Yarin Gal 외

Neural networks are extremely flexible models due to their large number of parameters, which is beneficial for learning, but also highly redundant. This makes it possible to compress neural networks without having a dras…

Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning

2021-10-22 · Soufiane Hayou, Bobby He, Gintare Karolina Dziugaite

We study an approach to learning pruning masks by optimizing the expected loss of stochastic pruning masks, i.e., masks which zero out each weight independently with some weight-specific probability. We analyze the train…

L2 Regularization regression

Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization

2025-08-12 · Gideon Vos, Liza van Eijk, Zoltan Sarnyai, Mostafa Rahimi Azghadi arxiv

Synaptic pruning in biological brains removes weak connections to improve efficiency. In contrast, dropout regularization in artificial neural networks randomly deactivates neurons without considering activity-dependent …

Time Series Forecasting