A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

2023-06-13 · Kihyuk Hong, Yuhang Li, Ambuj Tewari

Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained RL with general function approximation. PDCA runs a primal-dual algorithm on the Lagrangian function estimated by critics. The primal player employs a no-regret policy optimization oracle to maximize the Lagrangian estimate and the dual player acts greedily to minimize the Lagrangian estimate. We show that PDCA can successfully find a near saddle point of the Lagrangian, which is nearly optimal for the constrained RL problem. Unlike previous work that requires concentrability and a strong Bellman completeness assumption, PDCA only requires concentrability and realizability assumptions for sample-efficient learning.

📄 PDF Abstract BibTeX arXiv:2306.07818

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs

2024-02-07 · Kihyuk Hong, Ambuj Tewari

We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected data…

Offline RL Reinforcement Learning (RL)

Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes

2021-10-21 · Sihan Zeng, Thinh T. Doan, Justin Romberg

We consider a discounted cost constrained Markov decision process (CMDP) policy optimization problem, in which an agent seeks to maximize a discounted cumulative reward subject to a number of constraints on discounted cu…

Primal-dual Estimator Learning: an Offline Constrained Moving Horizon Estimation Method with Feasibility and Near-optimality Guarantees

2022-04-06 · Wenhan Cao, Jingliang Duan, Shengbo Eben Li, Chen Chen 외

This paper proposes a primal-dual framework to learn a stable estimator for linear constrained estimation problems leveraging the moving horizon approach. To avoid the online computational burden in most existing methods…

Offline Constrained RLHF with Multiple Preference Oracles

2026-03-31 · Brenden Latham, Mehrdad Moharrami arxiv

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population…

Reinforcement Learning

Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

2021-12-01 · NeurIPS 2021 12 · Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang

In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint. We focus on the offline setting where the RL agen…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)