Decoupled Context Processing for Context Augmented Language Modeling
Language models can be augmented with a context retriever to incorporate knowledge from large external databases. By leveraging retrieved context, the neural network does not have to memorize the massive amount of world knowledge within its internal parameters, leading to better parameter efficiency, interpretability and modularity. In this paper we examined a simple yet effective architecture for incorporating external context into language models based on decoupled Encoder Decoder architecture. We showed that such a simple architecture achieves competitive results on auto-regressive language modeling and open domain question answering tasks. We also analyzed the behavior of the proposed model which performs grounded context transfer. Finally we discussed the computational implications of such retrieval augmented models.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModelingLanguage ModellingOpen-Domain Question AnsweringQuestion AnsweringRetrievalWorld KnowledgeSimilar Papers 제목 키워드 기반
Augmenting Language Models with Long-Term Memory
Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework…
FormIn-Context LearningLanguage ModelingLanguage Modelling+1Crosslingual Retrieval Augmented In-context Learning for Bangla
The promise of Large Language Models (LLMs) in Natural Language Processing has often been overshadowed by their limited performance in low-resource languages such as Bangla. To address this, our paper presents a pioneeri…
In-Context LearningRetrievalMem-$π$: Adaptive Memory through Learning When and What to Generate
We present Mem-$π$, a framework for adaptive memory in large language model (LLM) agents, where useful guidance is generated on demand rather than retrieved from external memory stores. Existing memory-augmented agents t…
Reinforcement LearningEpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts
Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks. However, efficient processing of long contexts using LLMs remains a significant challenge. We introduce \textbf{Ep…
16kDecoderQuestion AnsweringRetrieval-augmented GenerationHMT: Hierarchical Memory Transformer for Long Context Language Processing
Transformer-based large language models (LLM) have been widely used in language processing applications. However, most of them restrict the context window that permits the model to attend to every token in the inputs. Pr…
Language ModelingLanguage ModellingMemorizationQuestion Answering