False Correlation Reduction for Offline Reinforcement Learning

2021-10-24 · Zhihong Deng, Zuyue Fu, Lingxiao Wang, Zhuoran Yang, Chenjia Bai, Tianyi Zhou, Zhaoran Wang, Jing Jiang

Offline reinforcement learning (RL) harnesses the power of massive datasets for resolving sequential decision problems. Most existing papers only discuss defending against out-of-distribution (OOD) actions while we investigate a broader issue, the false correlations between epistemic uncertainty and decision-making, an essential factor that causes suboptimality. In this paper, we propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL). The proposed algorithm introduces an annealing behavior cloning regularizer to help produce a high-quality estimation of uncertainty which is critical for eliminating false correlations from suboptimality. Theoretically, we justify the rationality of the proposed method and prove its convergence to the optimal policy with a sublinear rate under mild assumptions.

📄 PDF Abstract BibTeX arXiv:2110.12468

Code (1)

yifan123/arxiv_spider

Tasks

D4RL Decision Making Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?

2023-12-28 · Gunshi Gupta, Tim G. J. Rudner, Rowan Thomas McAllister, Adrien Gaidon 외

Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during training if most of the training data conta…

reinforcement-learning Reinforcement Learning

Horizon Reduction as Information Loss in Offline Reinforcement Learning

2025-12-25 · Uday Kumar Nidadala, Venkata Bhumika Guthi arxiv

Horizon reduction is a common design strategy in offline reinforcement learning (RL), used to mitigate long-horizon credit assignment, improve stability, and enable scalable learning through truncated rollouts, windowed …

Reinforcement Learning Offline RL

Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

2021-02-02 · NeurIPS 2021 12 · Ming Yin, Yu Bai, Yu-Xiang Wang

We consider the problem of offline reinforcement learning (RL) -- a well-motivated setting of RL that aims at policy optimization using only historical data. Despite its wide applicability, theoretical understandings of …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning

2023-08-28 · Hanhan Zhou, Tian Lan, Vaneet Aggarwal

Offline reinforcement learning aims to utilize datasets of previously gathered environment-action interaction records to learn a policy without access to the real environment. Recent work has shown that offline reinforce…

D4RL Off-policy evaluation OpenAI Gym reinforcement-learning+1

Adversarial Multi-Agent Reinforcement Learning for Proactive False Data Injection Detection

2024-11-19 · Kejun Chen, Truc Nguyen, Malik Hassanaly

Smart inverters are instrumental in the integration of renewable and distributed energy resources (DERs) into the electric grid. Such inverters rely on communication layers for continuous control and monitoring, potentia…

continuous-control Continuous Control Multi-agent Reinforcement Learning Transfer Learning