Improving Conditioning in Context-Aware Sequence to Sequence Models
Neural sequence to sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on cases where generation is conditioned on both a short query and a long context, such as abstractive question answering or document-level translation. We modify the standard sequence-to-sequence approach to make better use of both the query and the context by expanding the conditioning mechanism to intertwine query and context attention. We also introduce a simple and efficient data augmentation method for the proposed model. Experiments on three different tasks show that both changes lead to consistent improvements.
Code (0)
등록된 구현이 없습니다.
Tasks
abstractive question answeringData AugmentationOpen-Domain Question AnsweringQuestion AnsweringTranslationSimilar Papers 제목 키워드 기반
Diverse Pretrained Context Encodings Improve Document Translation
We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pretrained document context signals and assess the impact on translation performance of (1) different…
Document TranslationSentenceTranslationMIMIC: A Generative Multimodal Foundation Model for Biomolecules
Biological function emerges from coupled constraints across sequence, structure, regulation, evolution, and cellular context, yet most foundation models in biology are trained within one modality or for a fixed forward t…
Representation LearningPurposer: Putting Human Motion Generation in Context
We present a novel method to generate human motion to populate 3D indoor scenes. It can be controlled with various combinations of conditioning signals such as a path in a scene, target poses, past motions, and scenes re…
Motion GenerationRepresentation LearningKoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
Standard Large Language Model (LLM) pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to contextualize information. To bridge this …
SiDGen: Structure-informed Diffusion for Generative modeling of Ligands for Proteins
Structure-based drug design (SBDD) faces a fundamental scaling fidelity dilemma: rich pocket-aware conditioning captures interaction geometry but can be costly, often scales quadratically ($O(L^2)$) or worse with protein…