Document Context Language Models
Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.
Code (1)
Tasks
SentenceSimilar Papers 제목 키워드 기반
In-context Pretraining: Language Modeling Beyond Document Boundaries
Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. E…
In-Context LearningLanguage ModelingLanguage ModellingReading ComprehensionContext-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model
Although many context-aware neural machine translation models have been proposed to incorporate contexts in translation, most of those models are trained end-to-end on parallel documents aligned in sentence-level. Becaus…
DecoderLanguage ModelingLanguage ModellingMachine Translation+2Language Model Pre-training for Hierarchical Document Representations
Hierarchical neural architectures are often used to capture long-distance dependencies and have been applied to many document-level tasks such as summarization, document segmentation, and sentiment analysis. However, eff…
Document SummarizationExtractive Document SummarizationExtractive Text SummarizationLanguage Modeling+5Analyzing Context Utilization of LLMs in Document-Level Translation
Large language models (LLM) are increasingly strong contenders in machine translation. We study document-level translation, where some words cannot be translated without context from outside the sentence. We investigate …
Document TranslationMachine TranslationSentenceTranslationDocFinQA: A Long-Context Financial Reasoning Dataset
For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and data. Financial professionals often inte…
RetrievalSpecificity