Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning

2025-06-16 · Ali Baheri

Offline reinforcement learning promises policy improvement from logged interaction data alone, yet state-of-the-art algorithms remain vulnerable to value over-estimation and to violations of domain knowledge such as monotonicity or smoothness. We introduce implicit constraint-aware off-policy correction, a framework that embeds structural priors directly inside every Bellman update. The key idea is to compose the optimal Bellman operator with a proximal projection on a convex constraint set, which produces a new operator that (i) remains a $\gamma$-contraction, (ii) possesses a unique fixed point, and (iii) enforces the prescribed structure exactly. A differentiable optimization layer solves the projection; implicit differentiation supplies gradients for deep function approximators at a cost comparable to implicit Q-learning. On a synthetic Bid-Click auction -- where the true value is provably monotone in the bid -- our method eliminates all monotonicity violations and outperforms conservative Q-learning and implicit Q-learning in return, regret, and sample efficiency.

📄 PDF Abstract BibTeX arXiv:2506.14058

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction

2025-07-30 · Alex Durkin, Jasper Stolte, Matthew Jones, Raghuraman Pitchumani 외 arxiv

Offline reinforcement learning (offline RL) offers a promising framework for developing control strategies in chemical process systems using historical data, without the risks or costs of online experimentation. This wor…

Reinforcement Learning Offline RL

COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation

2022-04-19 · ICLR 2022 4 · Jongmin Lee, Cosmin Paduraru, Daniel J. Mankowitz, Nicolas Heess 외

We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collec…

Offline RL Off-policy evaluation reinforcement-learning Reinforcement Learning+1

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

2024-03-12 · Yunpeng Qing, Shunyu Liu, Jingyuan Cong, KaiXuan Chen 외

Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tac…

D4RL reinforcement-learning Reinforcement Learning (RL)

SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations

2023-09-21 · NeurIPS 2023 11

We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonst…

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

2021-06-07 · NeurIPS 2021 12 · Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng 외

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counte…

Multi-agent Reinforcement Learning Offline RL Q-Learning reinforcement-learning+3