Modeling Context With Linear Attention for Scalable Document-Level Translation
Document-level neural machine translation allows models to leverage dependencies beyond sentence-internal context to produce more coherent and consistent translations. However, these models, predominantly based on transformers, are difficult to scale to long documents due to the quadratic time and space complexity of their self-attention layers. Recent efforts on efficient attention variants improve scalability, but it is yet unclear if and to what extent their inductive biases are suitable for document translation. In this paper, we explore the efficacy of a recent linear attention model by Peng et al. (2021) on document-level translation and augment it with a sentential gating mechanism. We evaluate the model on the IWSLT 2015 and OpenSubtitles 2018 datasets against a strong transformer baseline and achieve up to 40% decoding speedup with similar or improved BLEU scores. We show that the sentential gate further improves translation quality on IWSLT, a dataset with long sequences.
Code (0)
등록된 구현이 없습니다.
Tasks
Document TranslationMachine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Modeling Context With Linear Attention for Scalable Document-Level Translation
Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations. However, these models, predominantly based on transformers, are difficult to scale to long do…
Document Level Machine TranslationDocument TranslationInductive BiasMachine Translation+2Scalable In-context Ranking with Generative Models
In-context Ranking (ICR) is an emerging paradigm for Information Retrieval (IR), which leverages contextual understanding of LLMs by directly incorporating the task description, candidate documents, and the query into th…
Information RetrievalBiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarization
Transformer-based architectures have advanced text summarization, yet their quadratic complexity limits scalability on long documents. This paper introduces BiSparse-AAS (Bilinear Sparse Attention with Adaptive Spans), a…
Text SummarizationSelective Attention for Context-aware Neural Machine Translation
Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentence…
DecoderMachine TranslationNMTSentence+1Log-Linear Attention
The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention a…
MambaState Space Models