Semantic HELM: A Human-Readable Memory for Reinforcement Learning
Reinforcement learning agents deployed in the real world often have to cope with partially observable environments. Therefore, most agents employ memory mechanisms to approximate the state of the environment. Recently, there have been impressive success stories in mastering partially observable environments, mostly in the realm of computer games like Dota 2, StarCraft II, or MineCraft. However, existing methods lack interpretability in the sense that it is not comprehensible for humans what the agent stores in its memory. In this regard, we propose a novel memory mechanism that represents past events in human language. Our method uses CLIP to associate visual inputs with language tokens. Then we feed these tokens to a pretrained language model that serves the agent as memory and provides it with a coherent and human-readable representation of the past. We train our memory mechanism on a set of partially observable environments and find that it excels on tasks that require a memory component, while mostly attaining performance on-par with strong baselines on tasks that do not. On a challenging continuous recognition task, where memorizing the past is crucial, our memory mechanism converges two orders of magnitude faster than prior methods. Since our memory mechanism is human-readable, we can peek at an agent's memory and check whether crucial pieces of information have been stored. This significantly enhances troubleshooting and paves the way toward more interpretable agents.
Code (1)
Tasks
Dota 2Language ModellingMinecraftreinforcement-learningReinforcement LearningStarcraftStarcraft IIMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generating Explanations from Deep Reinforcement Learning Using Episodic Memory
Deep Reinforcement Learning (RL) involves the use of Deep Neural Networks (DNNs) to make sequential decisions in order to maximize reward. For many tasks the resulting sequence of actions produced by a Deep RL policy can…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Large Language Models Do Not Always Need Readable Language
Large language models (LLMs) are commonly prompted and interfaced with human-readable natural language, even when the intended reader is another model. This paper investigates whether semantic information can be encoded …
PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agent
The development of autonomous tool-use agents for complex, long-horizon tasks in collaboration with human users has become the frontier of agentic research. During multi-turn Human-AI interactions, the dynamic and uncert…
Reinforcement LearningBeyond Human-Readable: Rethinking Software Engineering Conventions for the Agentic Development Era
For six decades, software engineering principles have been optimized for a single consumer: the human developer. The rise of agentic AI development, where LLM-based agents autonomously read, write, navigate, and debug co…
History Compression via Language Models in Reinforcement Learning
In a partially observable Markov decision process (POMDP), an agent typically uses a representation of the past to approximate the underlying MDP. We propose to utilize a frozen Pretrained Language Transformer (PLT) for …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)