Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

2021-12-01 · NeurIPS 2021 12 · Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang

In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint. We focus on the offline setting where the RL agent aims to learn the optimal policy from a given dataset. This scenario is common in real-world applications where interactions with the environment are expensive and the constraint violation is dangerous. For such a setting, we transform the original constrained problem into a primal-dual formulation, which is solved via dual gradient ascent. Moreover, we propose to combine such an approach with pessimism to overcome the uncertainty in offline data, which leads to our Pessimistic Dual Iteration (PEDI). We establish upper bounds on both the suboptimality and constraint violation for the policy learned by PEDI based on an arbitrary dataset, which proves that PEDI is provably sample efficient. We also specialize PEDI to the setting with linear function approximation. To the best of our knowledge, we propose the first provably efficient constrained multi-objective RL algorithm with offline data without any assumption on the coverage of the dataset.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-Objective Reinforcement Learningreinforcement-learning Reinforcement Learning (RL)

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27 · Fan Zhang, Baoru Huang, Xin Zhang arxiv

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…

Reinforcement Learning Offline RL

MOReL: Model-Based Offline Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. This serves as an extreme test for an agent's ability to e…

model Offline RL reinforcement-learning Reinforcement Learning+1

Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

2021-07-13 · ICLR 2022 4 · Masatoshi Uehara, Wen Sun

We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm named Constrained Pessimistic Policy Op…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning

2026-02-02 · Zeyu Fang, Zuyuan Zhang, Mahdi Imani, Tian Lan arxiv

Model-based offline reinforcement learning is brittle under distribution shift: policy improvement drives rollouts into state--action regions weakly supported by the dataset, where compounding model error yields severe v…

Reinforcement Learning

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning

2021-11-09 · Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan

Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)