Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning

2021-01-19 · Austin W. Hanjie, Victor Zhong, Karthik Narasimhan

We investigate the use of natural language to drive the generalization of control policies and introduce the new multi-task environment Messenger with free-form text manuals describing the environment dynamics. Unlike previous work, Messenger does not assume prior knowledge connecting text and state observations $-$ the control policy must simultaneously ground the game manual to entity symbols and dynamics in the environment. We develop a new model, EMMA (Entity Mapper with Multi-modal Attention) which uses an entity-conditioned attention module that allows for selective focus over relevant descriptions in the manual for each entity in the environment. EMMA is end-to-end differentiable and learns a latent grounding of entities and dynamics from text to observations using only environment rewards. EMMA achieves successful zero-shot generalization to unseen games with new dynamics, obtaining a 40% higher win rate compared to multiple baselines. However, win rate on the hardest stage of Messenger remains low (10%), demonstrating the need for additional work in this direction.

📄 PDF Abstract BibTeX arXiv:2101.07393

Code (1)

ahjwang/messenger-emma 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning (RL)Zero-shot Generalization

Entity Divider with Language Grounding in Multi-Agent Reinforcement Learning

2022-10-25 · Ziluo Ding, Wanpeng Zhang, Junpeng Yue, Xiangjun Wang 외

We investigate the use of natural language to drive the generalization of policies in multi-agent settings. Unlike single-agent settings, the generalization of policies should also consider the influence of other agents.…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Grounding Language to Entities for Generalization in Reinforcement Learning

2021-01-01 · H. J. Austin Wang, Karthik R Narasimhan

In this paper, we consider the problem of leveraging textual descriptions to improve generalization of control policies to new scenarios. Unlike prior work in this space, we do not assume access to any form of prior know…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Zero-shot Generalization

MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models

2025-11-23 · Xiyang Wu, Zongxia Li, Jihui Jin, Guangyao Shi 외 arxiv

Vision Language Models (VLMs) perform well on standard video tasks but struggle with physics-related reasoning involving motion dynamics and spatial interactions. We present a novel approach to address this gap by transl…

Visual Grounding

Zero-Shot Compositional Policy Learning via Language Grounding

2020-04-15 · Tianshi Cao, Jingkang Wang, Yining Zhang, Sivabalan Manivasagam

Despite recent breakthroughs in reinforcement learning (RL) and imitation learning (IL), existing algorithms fail to generalize beyond the training environments. In reality, humans can adapt to new tasks quickly by lever…

Descriptive Domain Adaptation Grounded language learning Imitation Learning+4

SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark

2021-10-20 · Victor Zhong, Austin W. Hanjie, Sida I. Wang, Karthik Narasimhan 외

Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding be…

Grounded language learning NetHack