Improving Autoregressive NLP Tasks via Modular Linearized Attention
Various natural language processing (NLP) tasks necessitate models that are efficient and small based on their ultimate application at the edge or in other resource-constrained environments. While prior research has reduced the size of these models, increasing computational efficiency without considerable performance impacts remains difficult, especially for autoregressive tasks. This paper proposes modular linearized attention (MLA), which combines multiple efficient attention mechanisms, including cosFormer, to maximize inference quality while achieving notable speedups. We validate this approach on several autoregressive NLP tasks, including speech-to-text neural machine translation (S2T NMT), speech-to-text simultaneous translation (SimulST), and autoregressive text-to-spectrogram, noting efficiency gains on TTS and competitive performance for NMT and SimulST during training and inference.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyMachine TranslationNMTSpeech-to-TextTranslationSimilar Papers 제목 키워드 기반
When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
Autoregressive Large Language Models (LLMs) have achieved impressive performance in language tasks but face two significant bottlenecks: (1) quadratic complexity in the attention module as the number of tokens increases,…
DecoderLeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions
A promising approach to preserving model performance in linearized transformers is to employ position-based re-weighting functions. However, state-of-the-art re-weighting functions rely heavily on target sequence lengths…
Language ModelingLanguage ModellingSimultaneous Speech-to-Text TranslationSpeech-to-Text+1Simple yet Effective Gradient-Free Graph Convolutional Networks
Linearized Graph Neural Networks (GNNs) have attracted great attention in recent years for graph representation learning. Compared with nonlinear Graph Neural Network (GNN) models, linearized GNNs are much more time-effi…
Graph Neural NetworkGraph Representation LearningNode ClassificationRepresentation LearningLinear attention is (maybe) all you need (to understand transformer optimization)
Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying …
AllLongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory
Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention varian…
Computational EfficiencyLanguage ModelingLanguage ModellingQuantization+1