Adversarial Model for Offline Reinforcement Learning

2023-02-21 · NeurIPS 2023 11 · Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang, Ching-An Cheng

We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with "any" admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.

📄 PDF Abstract BibTeX arXiv:2302.11048

Code (0)

등록된 구현이 없습니다.

Tasks

modelreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation

2025-03-26 · Hongye Cao, Fan Feng, Jing Huo, Shangdong Yang 외

Model-based offline Reinforcement Learning (RL) constructs environment models from offline datasets to perform conservative policy optimization. Existing approaches focus on learning state transitions through ensemble mo…

D4RL Data Augmentation Offline RL reinforcement-learning+2

Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses

2024-05-18 · Thanh Nguyen, Tung M. Luu, Tri Ton, Chang D. Yoo

Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning i…

D4RL Offline RL Reinforcement Learning (RL)

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

2022-06-06 · Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang 외

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are gener…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

2022-04-26 · Marc Rigter, Bruno Lacerda, Nick Hawes

Offline reinforcement learning (RL) aims to find performant policies from logged data without further environment interaction. Model-based algorithms, which learn a model of the environment from the dataset and perform c…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

2019-12-01 · NeurIPS 2019 12 · Xueying Bai, Jian Guan, Hongning Wang

Reinforcement learning is effective in optimizing policies for recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with a real environment, and thus are expen…

Generative Adversarial Network Model-based Reinforcement Learning Recommendation Systems reinforcement-learning+2