MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards

2026-01-09 · Zhiyu Shen, Ziming Wu, Fuming Lai, Shaobing Lian, Yanghui Rao arxiv

Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states. While memory-augmented frameworks offer a structured alternative, current systems rely on static prompting of closed-source models or suffer from ineffective training paradigms with sparse rewards. We introduce MemBuilder, a reinforcement learning framework that trains models to orchestrate multi-dimensional memory construction with attributed dense rewards. MemBuilder addresses two key challenges: (1) Sparse Trajectory-Level Rewards: we employ synthetic session-level question generation to provide dense intermediate rewards across extended trajectories; and (2) Multi-Dimensional Memory Attribution: we introduce contribution-aware gradient weighting that scales policy updates based on each component's downstream impact. Experimental results show that MemBuilder enables a 4B-parameter model to outperform state-of-the-art closed-source baselines, exhibiting strong generalization across long-term dialogue benchmarks.

📄 PDF Abstract BibTeX arXiv:2601.05488

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Question Generation

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

2024-10-14 · Hung Le, Kien Do, Dung Nguyen, Sunil Gupta 외

Effective decision-making in partially observable environments demands robust memory management. Despite their success in supervised learning, current deep-learning memory models struggle in reinforcement learning enviro…

Decision Making Management Meta Reinforcement Learning reinforcement-learning+1

From prosthetic memory to prosthetic denial: Auditing whether large language models are prone to mass atrocity denialism

2025-05-27 · Roberto Ulloa, Eve M. Zucker, Daniel Bultmann, David J. Simon 외

The proliferation of large language models (LLMs) can influence how historical narratives are disseminated and perceived. This study explores the implications of LLMs' responses on the representation of mass atrocity mem…

PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

2026-02-01 · Sidharth Pulipaka, Oliver Chen, Manas Sharma, Taaha S Bajwa 외 arxiv

Conversational assistants are increasingly integrating long-term memory with large language models (LLMs). This persistence of memories, e.g., the user is vegetarian, can enhance personalization in future conversations. …

Augmenting Language Models with Long-Term Memory

2023-06-12 · NeurIPS 2023 11 · Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu 외

Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework…

Form In-Context Learning Language Modeling Language Modelling+1

Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections

2026-02-17 · Xianglin Yang, Yufei He, Shuo Ji, Bryan Hooi 외 arxiv

Self-evolving LLM agents update their internal state across sessions, often by writing and reusing long-term memory. This design improves performance on long-horizon tasks but creates a security risk: untrusted external …