Diverse Pretrained Context Encodings Improve Document Translation
We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pretrained document context signals and assess the impact on translation performance of (1) different pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts. Experiments on the NIST Chinese-English, and IWSLT and WMT English-German tasks support four general conclusions: that using pretrained context representations markedly improves sample efficiency, that adequate parallel data resources are crucial for learning to use document context, that jointly conditioning on multiple context representations outperforms any single representation, and that source context is more valuable for translation performance than target side context. Our best multi-context model consistently outperforms the best existing context-aware transformers.
Code (0)
등록된 구현이 없습니다.
Tasks
Document TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification
Long document classification presents challenges in capturing both local and global dependencies due to their extensive content and complex structure. Existing methods often struggle with token limits and fail to adequat…
Document ClassificationGraph AttentionSentenceDeep Relevance Ranking Using Enhanced Document-Query Interactions
We explore several new models for document relevance ranking, building upon the Deep Relevance Matching Model (DRMM) of Guo et al. (2016). Unlike DRMM, which uses context-insensitive encodings of terms and query-document…
Ad-Hoc Information RetrievalQuestion AnsweringDIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization
Identifying relevant knowledge to be used in conversational systems that are grounded in long documents is critical to effective response generation. We introduce a knowledge identification model that leverages the docum…
Response GenerationPerceptually Aligning Representations of Music via Noise-Augmented Autoencoders
We argue that training autoencoders to reconstruct inputs from noised versions of their encodings, when combined with perceptually motivated losses, yields encodings that are structured according to a perceptual hierarch…
Task Matrices: Linear Maps for Cross-Model Finetuning Transfer
Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more g…