Relational Memory Augmented Language Models
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model with a knowledge graph for a more coherent and logical generation.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingText GenerationSimilar Papers 제목 키워드 기반
Distributed Associative Memory Network with Association Reinforcing Loss
Despite recent progress in memory augmented neural network research, associative memory networks with a single external memory still show limited performance on complex relational reasoning tasks. The main reason for thi…
MemorizationRelational ReasoningRelational Database Augmented Large Language Model
Large language models (LLMs) excel in many natural language processing (NLP) tasks. However, since LLMs can only incorporate new knowledge through training or supervised fine-tuning processes, they are unsuitable for app…
Language ModelingLanguage ModellingLarge Language ModelmodelMemory-Augmented Dynamic Neural Relational Inference
Dynamic interacting systems are prevalent in vision tasks. These interactions are usually difficult to observe and measure directly, and yet understanding latent interactions is essential for performing inference tas…
Trajectory PredictionSARN: Relational Reasoning through Sequential Attention
This paper proposes an attention module augmented relational network called SARN(Sequential Attention Relational Network) that can carry out relational reasoning by extracting reference objects and making efficient pairi…
Relational ReasoningSignificant Other AI: Identity, Memory, and Emotional Regulation as Long-Term Relational Intelligence
Significant Others (SOs) stabilize identity, regulate emotion, and support narrative meaning-making, yet many people today lack access to such relational anchors. Recent advances in large language models and memory-augme…