Attending to Long-Distance Document Context for Sequence Labeling
We present in this work a method for incorporating global context in long documents when making local decisions in sequence labeling problems like NER. Inspired by work in featurized log-linear models (Chieu and Ng, 2002; Sutton and McCallum, 2004), our model learns to attend to multiple mentions of the same word type in generating a representation for each token in context, extending that work to learning representations that can be incorporated into modern neural models. Attending to broader context at test time provides complementary information to pretraining (Gururangan et al., 2020), yields strong gains over equivalently parameterized models lacking such context, and performs best at recognizing entities with high TF-IDF scores (i.e., those that are important within a document).
Code (1)
Tasks
NERSimilar Papers 제목 키워드 기반
Coarse-to-Fine Attention Models for Document Summarization
Sequence-to-sequence models with attention have been successful for a variety of NLP problems, but their speed does not scale well for tasks with long source sequences such as document summarization. We propose a novel c…
Document SummarizationMachine TranslationQuestion AnsweringImproving Long Context Document-Level Machine Translation
Document-level context for neural machine translation (NMT) is crucial to improve the translation consistency and cohesion, the translation of ambiguous inputs, as well as several other linguistic phenomena. Many works h…
Document Level Machine TranslationMachine TranslationNMTSentence+1Unlimiformer: Long-Range Transformers with Unlimited Length Input
Since the proposal of transformers, these models have been limited to bounded input lengths, because of their need to attend to every token in the input. In this work, we propose Unlimiformer: a general approach that wra…
Book summarizationCPUDecoderGPUUntie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts presents significant challenges. These includ…
Data AugmentationLoong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selection
Document-level translation remains one of the most challenging tasks for large language models, which are constrained by limited context windows that impede global cohesion, while simultaneously suffering from redundant …
Reinforcement Learning