Offline Reinforcement Learning with Resource Constrained Online Deployment

2021-09-29 · Jayanth Reddy Regatti, Aniket Anand Deshmukh, Young Hun Jung, Frank Cheng, Abhishek Gupta, Urun Dogan

Offline reinforcement learning is used to train policies in scenarios where real-time access to the environment is expensive or impossible. As a natural consequence of these harsh conditions, an agent may lack the resources to fully observe the online environment before taking an action. We dub this situation the resource-constrained setting. This leads to situations where the offline dataset (available for training) can contain fully processed features (using powerful language models, image models, complex sensors, etc.) which are not available when actions are actually taken online. This disconnect leads to an interesting and unexplored problem in offline RL: Is it possible to use a richly processed offline dataset to train a policy which has access to fewer features in the online environment? In this work, we introduce and formalize this novel resource-constrained problem setting. We highlight the performance gap between policies trained using the full offline dataset and policies trained using limited features. We address this performance gap with a policy transfer algorithm which first trains a teacher agent using the offline dataset where features are fully available, and then transfers this knowledge to a student agent that only uses the resource-constrained features. To better capture the challenge of this setting, we propose a data collection procedure: Resource Constrained-Datasets for RL (RC-D4RL). We evaluate our transfer algorithm on RC-D4RL and the popular D4RL benchmarks and observe consistent improvement over the baseline (TD3+BC without transfer).

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline RL With Resource Constrained Online Deployment

2021-10-07 · Jayanth Reddy Regatti, Aniket Anand Deshmukh, Frank Cheng, Young Hun Jung 외

D4RL Offline RL

MURO: Deployment Constrained Reinforcement Learning with Model-based Uncertainty Regularized Batch Optimization

2021-09-29 · DiJia Su, Jason D. Lee, John Mulvey, H. Vincent Poor

In many contemporary applications such as healthcare, finance, robotics, and recommendation systems, continuous deployment of new policies for data collection and online learning is either cost ineffective or impractical…

Recommendation Systems reinforcement-learning Reinforcement Learning (RL)Uncertainty Quantification

MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning

2021-02-23 · DiJia Su, Jason D. Lee, John M. Mulvey, H. Vincent Poor

Reinforcement Learning (RL)Uncertainty Quantification

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

2023-06-13 · Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang 외

Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often…

D4RL Efficient Exploration Offline RL reinforcement-learning+1

FOSP: Fine-tuning Offline Safe Policy through World Models

2024-07-06 · Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He 외

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize …

Model-based Reinforcement Learning Offline RL Reinforcement Learning (RL)Safe Reinforcement Learning