Policy Constraint by Only Support Constraint for Offline Reinforcement Learning

2025-03-07 · Yunkai Gao, Jiaming Guo, Fan Wu, Rui Zhang

Offline reinforcement learning (RL) aims to optimize a policy by using pre-collected datasets, to maximize cumulative rewards. However, offline reinforcement learning suffers challenges due to the distributional shift between the learned and behavior policies, leading to errors when computing Q-values for out-of-distribution (OOD) actions. To mitigate this issue, policy constraint methods aim to constrain the learned policy's distribution with the distribution of the behavior policy or confine action selection within the support of the behavior policy. However, current policy constraint methods tend to exhibit excessive conservatism, hindering the policy from further surpassing the behavior policy's performance. In this work, we present Only Support Constraint (OSC) which is derived from maximizing the total probability of learned policy in the support of behavior policy, to address the conservatism of policy constraint. OSC presents a regularization term that only restricts policies to the support without imposing extra constraints on actions within the support. Additionally, to fully harness the performance of the new policy constraints, OSC utilizes a diffusion model to effectively characterize the support of behavior policies. Experimental evaluations across a variety of offline RL benchmarks demonstrate that OSC significantly enhances performance, alleviating the challenges associated with distributional shifts and mitigating conservatism of policy constraints. Code is available at https://github.com/MoreanP/OSC.

📄 PDF Abstract BibTeX arXiv:2503.05207

Code (1)

moreanp/osc 공식 구현 pytorch

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning

2025-11-04 · Yixiu Mao, Yun Qu, Qi Wang, Xiangyang Ji arxiv

Offline reinforcement learning (RL) suffers from extrapolation errors induced by out-of-distribution (OOD) actions. To address this, offline RL algorithms typically impose constraints on action selection, which can be sy…

Reinforcement Learning Bilevel Optimization Offline RL

Offline Model-based Adaptable Policy Learning

2021-12-01 · NeurIPS 2021 12 · Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo 외

In reinforcement learning, a promising direction to avoid online trial-and-error costs is learning from an offline dataset. Current offline reinforcement learning methods commonly learn in the policy space constrained to…

Decision Making model MuJoCo reinforcement-learning+2

Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning

2024-12-11 · Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, Cody Fleming

In safe offline reinforcement learning (RL), the objective is to develop a policy that maximizes cumulative rewards while strictly adhering to safety constraints, utilizing only offline data. Traditional methods often fa…

Autonomous Driving Offline RL reinforcement-learning Reinforcement Learning+2

ReDS: Offline RL With Heteroskedastic Datasets via Support Constraints

2023-09-21 · NeurIPS 2023 11

Offline reinforcement learning (RL) learns policies entirely from static datasets. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors chang…

Supported Policy Optimization for Offline Reinforcement Learning

2022-02-13 · Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang 외

Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy. The elabo…