Policy-regularized Offline Multi-objective Reinforcement Learning

2024-01-04 · Qian Lin, Chao Yu, Zongkai Liu, Zifan Wu

In this paper, we aim to utilize only offline trajectory data to train a policy for multi-objective RL. We extend the offline policy-regularized method, a widely-adopted approach for single-objective offline RL problems, into the multi-objective setting in order to achieve the above goal. However, such methods face a new challenge in offline MORL settings, namely the preference-inconsistent demonstration problem. We propose two solutions to this problem: 1) filtering out preference-inconsistent demonstrations via approximating behavior preferences, and 2) adopting regularization techniques with high policy expressiveness. Moreover, we integrate the preference-conditioned scalarized update method into policy-regularized offline RL, in order to simultaneously learn a set of policies using a single policy network, thus reducing the computational cost induced by the training of a large number of individual policies for various preferences. Finally, we introduce Regularization Weight Adaptation to dynamically determine appropriate regularization weights for arbitrary target preferences during deployment. Empirical results on various multi-objective datasets demonstrate the capability of our approach in solving offline MORL problems.

📄 PDF Abstract BibTeX arXiv:2401.02244

Code (1)

qianlin04/prmorl 공식 구현 pytorch

Tasks

Multi-Objective Reinforcement LearningOffline RL reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning

2021-10-02 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor K Prasanna

Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Uncertainty Regularized Policy Learning for Offline Reinforcement Learning

2021-09-29 · Han Zheng, Jing Jiang, Pengfei Wei, Guodong Long 외

Recent studies show the promising results of using online RL methods in the offline setting. However, such a learning diagram may suffer from an overtraining issue, that is, the performance of the policy degrades signifi…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Robust Regularized Policy Iteration under Transition Uncertainty

2026-03-10 · Hongqiang Lin, Zhenghui Fu, Weihao Tang, Pengfei Wang 외 arxiv

Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distributio…

Reinforcement Learning Offline RL

Offline Reinforcement Learning with Soft Behavior Regularization

2021-10-14 · Haoran Xu, Xianyuan Zhan, Jianxiong Li, Honglei Yin

Most prior approaches to offline reinforcement learning (RL) utilize \textit{behavior regularization}, typically augmenting existing off-policy actor critic algorithms with a penalty measuring divergence between the poli…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Conservative Offline Policy Adaptation in Multi-Agent Games

2023-09-21 · NeurIPS 2023 11

Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progre…