Pessimistic Off-Policy Multi-Objective Optimization

2023-10-28 · Shima Alizadeh, Aniruddha Bhargava, Karthick Gopalswamy, Lalit Jain, Branislav Kveton, Ge Liu

Multi-objective optimization is a type of decision making problems where multiple conflicting objectives are optimized. We study offline optimization of multi-objective policies from data collected by an existing policy. We propose a pessimistic estimator for the multi-objective policy values that can be easily plugged into existing formulas for hypervolume computation and optimized. The estimator is based on inverse propensity scores (IPS), and improves upon a naive IPS estimator in both theory and experiments. Our analysis is general, and applies beyond our IPS estimators and methods for optimizing them. The pessimistic estimator can be optimized by policy gradients and performs well in all of our experiments.

📄 PDF Abstract BibTeX arXiv:2310.18617

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization

2023-11-10 · Jared Markowitz, Edward W. Staley

To facilitate efficient learning, policy gradient approaches to deep reinforcement learning (RL) are typically paired with variance reduction measures and strategies for making large but safe policy changes based on a ba…

Deep Reinforcement Learning Multi-Task Learning Policy Gradient Methods Reinforcement Learning (RL)

You May Not Need Ratio Clipping in PPO

2022-01-31 · Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin 외

Proximal Policy Optimization (PPO) methods learn a policy by iteratively performing multiple mini-batch optimization epochs of a surrogate objective with one set of sampled data. Ratio clipping PPO is a popular variant t…

continuous-control Continuous Control

Mitigating Mismatch within Reference-based Preference Optimization

2026-02-12 · Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng 외 arxiv

Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models, but its reliance on a reference policy introduces a critical tension. DPO weighs each updat…

POPO: Pessimistic Offline Policy Optimization

2020-12-26 · Qiang He, Xinwen Hou

Offline reinforcement learning (RL), also known as batch RL, aims to optimize policy from a large pre-recorded dataset without interaction with the environment. This setting offers the promise of utilizing diverse, pre-c…

Offline RL Q-Learning Reinforcement Learning (RL)

Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

2021-12-01 · NeurIPS 2021 12 · Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang

In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint. We focus on the offline setting where the RL agen…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)