Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning

2022-07-19 · Wenhao Ding, Haohong Lin, Bo Li, Ding Zhao

As a pivotal component to attaining generalizable solutions in human intelligence, reasoning provides great potential for reinforcement learning (RL) agents' generalization towards varied goals by summarizing part-to-whole arguments and discovering cause-and-effect relations. However, how to discover and represent causalities remains a huge gap that hinders the development of causal RL. In this paper, we augment Goal-Conditioned RL (GCRL) with Causal Graph (CG), a structure built upon the relation between objects and events. We novelly formulate the GCRL problem into variational likelihood maximization with CG as latent variables. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventional data to estimate the posterior of CG; using CG to learn generalizable models and interpretable policies. Due to the lack of public benchmarks that verify generalization capability under reasoning, we design nine tasks and then empirically show the effectiveness of the proposed method against five baselines on these tasks. Further theoretical analysis shows that our performance improvement is attributed to the virtuous cycle of causal discovery, transition modeling, and policy training, which aligns with the experimental evidence in extensive ablation studies.

📄 PDF Abstract BibTeX arXiv:2207.09081

Code (1)

gilgameshd/grader 공식 구현 pytorch

Tasks

Causal Discoveryreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Inverse Reinforcement Learning with Natural Language Goals

2020-08-16 · Li Zhou, Kevin Small

Humans generally use natural language to communicate task requirements to each other. Ideally, natural language should also be usable for communicating goals to autonomous machines (e.g., robots) to minimize friction in …

Friction Instruction Following reinforcement-learning Reinforcement Learning+1

Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning

2025-02-11 · Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang 외

State-of-the-art (SOTA) reinforcement learning (RL) methods have enabled vision-language model (VLM) agents to learn from interaction with online environments without human supervision. However, these methods often strug…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning Generative Models with Goal-conditioned Reinforcement Learning

2023-03-26 · Mariana Vargas Vieyra, Pierre Ménard

We present a novel, alternative framework for learning generative models with goal-conditioned reinforcement learning. We define two agents, a goal conditioned agent (GC-agent) and a supervised agent (S-agent). Given a u…

Image Generation reinforcement-learning Reinforcement Learning

Improvements on Hindsight Learning

2018-09-16 · Ameet Deshpande, Srikanth Sarma, Ashutosh Jha, Balaraman Ravindran

Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models

2023-10-30 · Mianchu Wang, Rui Yang, Xi Chen, Hao Sun 외

Offline Goal-Conditioned RL (GCRL) offers a feasible paradigm for learning general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods, mai…

Generative Adversarial Network reinforcement-learning