Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

2020-06-01 · Anoopkumar Sonar, Vincent Pacelli, Anirudha Majumdar

A fundamental challenge in reinforcement learning is to learn policies that generalize beyond the operating domains experienced during training. In this paper, we approach this challenge through the following invariance principle: an agent must find a representation such that there exists an action-predictor built on top of this representation that is simultaneously optimal across all training domains. Intuitively, the resulting invariant policy enhances generalization by finding causes of successful actions. We propose a novel learning algorithm, Invariant Policy Optimization (IPO), that implements this principle and learns an invariant policy during training. We compare our approach with standard policy gradient methods and demonstrate significant improvements in generalization performance on unseen domains for linear quadratic regulator and grid-world problems, and an example where a robot must learn to open doors with varying physical properties.

📄 PDF Abstract BibTeX arXiv:2006.01096

Code (1)

irom-lab/Invariant-Policy-Optimization 공식 구현 pytorch

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Proximal Supervised Fine-Tuning

2025-08-25 · Wenhong Zhu, Ruobing Xie, Rui Wang, Xingwu Sun 외 arxiv

Supervised fine-tuning (SFT) of foundation models often leads to poor generalization, where prior capabilities deteriorate after tuning on new tasks or domains. Inspired by trust-region policy optimization (TRPO) and pro…

Reinforcement Learning Domain Generalization

Decoupling Value and Policy for Generalization in Reinforcement Learning

2021-02-20 · Roberta Raileanu, Rob Fergus

Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that more information is needed to accuratel…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

2021-05-18 · Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang 외

Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents. To address such a chall…

Inductive Bias Multi-agent Reinforcement Learning

WMPO: World Model-based Policy Optimization for Vision-Language-Action Models

2025-11-12 · Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou 외 arxiv

Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation, but their reliance on expert demonstrations limits their ability to learn from failures and perform self-correctio…

Reinforcement Learning

A Principled Permutation Invariant Approach to Mean-Field Multi-Agent Reinforcement Learning

2021-09-29 · Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang 외

Inductive Bias Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)