Long Context Modeling with Ranked Memory-Augmented Retrieval
Effective long-term memory management is crucial for language models handling extended contexts. We introduce a novel framework that dynamically ranks memory entries based on relevance. Unlike previous works, our model introduces a novel relevance scoring and a pointwise re-ranking model for key-value embeddings, inspired by learning-to-rank techniques in information retrieval. Enhanced Ranked Memory Augmented Retrieval ERMAR achieves state-of-the-art results on standard benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalLearning-To-RankManagementRe-RankingRetrievalSimilar Papers 제목 키워드 기반
Augmenting Language Models with Long-Term Memory
Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework…
FormIn-Context LearningLanguage ModelingLanguage Modelling+1MemLong: Memory-Augmented Retrieval for Long Text Modeling
Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space comp…
4kDecoderGPUInformation Retrieval+4Knowledge Graph Enhanced Memory-Augmented Retrieval for Long Context Modeling
Long-context language modeling requires not only extending context windows but maintaining coherent understanding of entity states and relationships across thousands of tokens -- a challenge that semantic similarity alon…
Semantic SimilarityRelation ExtractionKnowledge GraphsStateful Memory-Augmented Transformers for Efficient Dialogue Modeling
Transformer encoder-decoder models have achieved great performance in dialogue generation tasks, however, their inability to process long dialogue history often leads to truncation of the context To address this problem,…
DecoderDialogue GenerationLanguage ModelingLanguage ModellingNTUA-SLP at SemEval-2018 Task 2: Predicting Emojis using RNNs with Context-aware Attention
In this paper we present a deep-learning model that competed at SemEval-2018 Task 2 "Multilingual Emoji Prediction". We participated in subtask A, in which we are called to predict the most likely associated emoji in Eng…
Task 2Word Embeddings