Bayesian Sparsification of Recurrent Neural Networks

2017-07-31 · Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov

Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights. Recently proposed Sparse Variational Dropout eliminates the majority of the weights in a feed-forward neural network without significant loss of quality. We apply this technique to sparsify recurrent neural networks. To account for recurrent specifics we also rely on Binary Variational Dropout for RNN. We report 99.5% sparsity level on sentiment analysis task without a quality drop and up to 87% sparsity level on language modeling task with slight loss of accuracy.

📄 PDF Abstract BibTeX arXiv:1708.00077

Code (2)

tipt0p/SparseBayesianRNN 공식 구현

ars-ashuha/variational-dropout-sparsifies-dnn tf

Tasks

Language ModelingLanguage Modelling Sentiment Analysis

Methods 이 논문이 사용한 방법론

Variational Dropout Variational Dropout is a regularization technique based on dropout, but uses a variational inference grounded approach. In…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Bayesian Sparsification of Gated Recurrent Neural Networks

2018-12-12 · NIPS Workshop CDNNRIA 2018 · Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov

Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent archit…

Bayesian Compression for Natural Language Processing

2018-10-25 · EMNLP 2018 10 · Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov

In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the …

Novel Gradient Sparsification Algorithm via Bayesian Inference

2024-09-23 · Ali Bereyhi, Ben Liang, Gary Boudreau, Ali Afana

Error accumulation is an essential component of the Top-$k$ sparsification method in distributed gradient descent. It implicitly scales the learning rate and prevents the slow-down of lateral movement, but it can also de…

Bayesian Inference

Structured Sparsification of Gated Recurrent Neural Networks

2019-11-13 · Ekaterina Lobacheva, Nadezhda Chirkova, Alexander Markovich, Dmitry Vetrov

Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent archi…

Language Modeling Language Modelling text-classification Text Classification

Bayesian sparsification for deep neural networks with Bayesian model reduction

2023-09-21 · Dimitrije Marković, Karl J. Friston, Stefan J. Kiebel

Deep learning's immense capabilities are often constrained by the complexity of its models, leading to an increasing demand for effective sparsification techniques. Bayesian sparsification for deep learning emerges as a …

Deep Learning Variational Inference