Efficient Language Modeling with Automatic Relevance Determination in Recurrent Neural Networks
Reduction of the number of parameters is one of the most important goals in Deep Learning. In this article we propose an adaptation of Doubly Stochastic Variational Inference for Automatic Relevance Determination (DSVI-ARD) for neural networks compression. We find this method to be especially useful in language modeling tasks, where large number of parameters in the input and output layers is often excessive. We also show that DSVI-ARD can be applied together with encoder-decoder weight tying allowing to achieve even better sparsity and performance. Our experiments demonstrate that more than 90{\%} of the weights in both encoder and decoder layers can be removed with a minimal quality loss.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModelingLanguage ModellingVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Laplacian Prior Variational Automatic Relevance Determination for Transmission Tomography
In the classic sparsity-driven problems, the fundamental L-1 penalty method has been shown to have good performance in reconstructing signals for a wide range of problems. However this performance relies on a good choice…
An Automatic Relevance Determination Prior Bayesian Neural Network for Controlled Variable Selection
We present an Automatic Relevance Determination prior Bayesian Neural Network(BNN-ARD) weight l2-norm measure as a feature importance statistic for the model-x knockoff filter. We show on both simulated data and the Norw…
Feature ImportanceVariable SelectionVariable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution
Variable selection for Gaussian process models is often done using automatic relevance determination, which uses the inverse length-scale parameter of each input variable as a proxy for variable relevance. This implicitl…
Gaussian ProcessesSensitivityVariable SelectionAutomatic Relevance Determination For Deep Generative Models
A recurring problem when building probabilistic latent variable models is regularization and model selection, for instance, the choice of the dimensionality of the latent space. In the context of belief networks with lat…
Model SelectionVariational InferenceCorrentropy-Based Logistic Regression with Automatic Relevance Determination for Robust Sparse Brain Activity Decoding
Recent studies have utilized sparse classifications to predict categorical variables from high-dimensional brain activity signals to expose human's intentions and mental states, selecting the relevant features automatica…
Brain DecodingClassificationEEGElectroencephalogram (EEG)+1