Bayesian Compression for Natural Language Processing
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable. Code is available on github: https://github.com/tipt0p/SparseBayesianRNN
Code (3)
Similar Papers 제목 키워드 기반
Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
In recent years, large language models (LLMs) have driven advances in natural language processing. Still, their growing scale has increased the computational burden, necessitating a balance between efficiency and perform…
Bayesian OptimizationLow-rank compressionFrom Imitation to Prediction, Data Compression vs Recurrent Neural Networks for Natural Language Processing
In recent studies [1][13][12] Recurrent Neural Networks were used for generative processes and their surprising performance can be explained by their ability to create good predictions. In addition, data compression is a…
Data CompressionThe SP theory of intelligence: an overview
This article is an overview of the "SP theory of intelligence". The theory aims to simplify and integrate concepts across artificial intelligence, mainstream computing and human perception and cognition, with information…
RetrievalBayesian Kernel Methods for Natural Language Processing
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing
Recent work raises concerns about the use of standard splits to compare natural language processing models. We propose a Bayesian statistical model comparison technique which uses k-fold cross-validation across multiple …