DictFormer: Tiny Transformer with Shared Dictionary
We introduce DictFormer with efficient shared dictionary to provide a compact, fast, and accurate transformer model. DictFormer significantly reduces the redundancy in the transformer's parameters by replacing the prior transformer's parameters with compact, shared dictionary, a few unshared coefficients, and indices. Also, DictFormer enables faster computations since expensive weights multiplications are converted into cheap shared look-ups on dictionary and a few linear projections. Training dictionary and coefficients are not trivial since indices used for looking up dictionary are not differentiable. We adopt a sparse-constraint training with $l_1\,\,norm$ relaxation to learn coefficients and indices in DictFormer. DictFormer is flexible to support different model sizes by dynamically changing dictionary size. Compared to existing lightweight Transformers, DictFormer consistently improves performance on multiple tasks, e.g., machine translation, abstractive summarization, and language modeling. Extensive experiments show that DictFormer outperforms prior light-weight transformer by $\sim2$ BLEU for machine translation task and achieves $\sim 1.7$ lower perplexity for the language modeling task, when matching or reducing the model size of the original transformer. Also, DictFormer reduces $6.3\times$ to $8.9\times$ model size on average without losing accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationLanguage ModelingLanguage ModellingMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning
Large language models have revolutionized AI applications, yet their high computational and memory demands hinder their widespread deployment. Existing compression techniques focus on intra-block optimizations (e.g., low…
Image ClassificationDNF: Unconditional 4D Generation with Dictionary-based Neural Fields
While remarkable success has been achieved through diffusion-based 3D generative models for shapes, 4D generative modeling remains challenging due to the complexity of object deformations over time. We propose DNF, a new…
Dictionary LearningClass specific or shared? A cascaded dictionary learning framework for image classification
Dictionary learning methods can be split into: i) class specific dictionary learning ii) class shared dictionary learning. The difference between the two categories is how to use discriminative information. With the firs…
Dictionary LearningGeneral Classificationimage-classificationImage ClassificationOn The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation
This work investigates the alignment problem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by augmen…
Machine TranslationTranslationCOMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dicti…