Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning

2023-11-25 · Melrose Roderick, Gaurav Manek, Felix Berkenkamp, J. Zico Kolter

A key problem in off-policy Reinforcement Learning (RL) is the mismatch, or distribution shift, between the dataset and the distribution over states and actions visited by the learned policy. This problem is exacerbated in the fully offline setting. The main approach to correct this shift has been through importance sampling, which leads to high-variance gradients. Other approaches, such as conservatism or behavior-regularization, regularize the policy at the cost of performance. In this paper, we propose a new approach for stable off-policy Q-Learning. Our method, Projected Off-Policy Q-Learning (POP-QL), is a novel actor-critic algorithm that simultaneously reweights off-policy samples and constrains the policy to prevent divergence and reduce value-approximation error. In our experiments, POP-QL not only shows competitive performance on standard benchmarks, but also out-performs competing methods in tasks where the data-collection policy is significantly sub-optimal.

📄 PDF Abstract BibTeX arXiv:2311.14885

Code (0)

등록된 구현이 없습니다.

Tasks

Q-LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

2023-06-06 · Linjie Xu, Zhengyao Jiang, Jinyu Wang, Lei Song 외

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) a…

D4RL MuJoCo Offline RL reinforcement-learning+2

Projected State-action Balancing Weights for Offline Reinforcement Learning

2021-09-10 · Jiayi Wang, Zhengling Qi, Raymond K. W. Wong

Offline policy evaluation (OPE) is considered a fundamental and challenging problem in reinforcement learning (RL). This paper focuses on the value estimation of a target policy based on pre-collected data generated from…

Causal Inference reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Data-Based Efficient Off-Policy Stabilizing Optimal Control Algorithms for Discrete-Time Linear Systems via Damping Coefficients

2024-12-30 · Dongdong Li, Jiuxiang Dong

Policy iteration is one of the classical frameworks of reinforcement learning, which requires a known initial stabilizing control. However, finding the initial stabilizing control depends on the known system model. To re…

Q-Learning reinforcement-learning Reinforcement Learning

Trust Region Q Adjoint Matching

2026-05-26 · Yonghoon Dong, Kyungmin Lee, Changyeon Kim, Jaehyuk Kim 외 arxiv

Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addr…

Reinforcement Learning Offline RL

Stability-Constrained Markov Decision Processes Using MPC

2021-02-02 · Mario Zanon, Sébastien Gros, Michele Palladino

In this paper, we consider solving discounted Markov Decision Processes (MDPs) under the constraint that the resulting policy is stabilizing. In practice MDPs are solved based on some form of policy approximation. We wil…

Model Predictive Control