To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

2024-07-01 · Tao Ma, Xuzhi Yang, Zoltan Szabo

Reinforcement learning (RL) -- finding the optimal behaviour (also referred to as policy) maximizing the collected long-term cumulative reward -- is among the most influential approaches in machine learning with a large number of successful applications. In several decision problems, however, one faces the possibility of policy switching -- changing from the current policy to a new one -- which incurs a non-negligible cost, and in the decision one is limited to using historical data without the availability for further online interaction. Despite the inevitable importance of this offline learning scenario, to our best knowledge, very little effort has been made to tackle the key problem of balancing between the gain and the cost of switching in a flexible and principled way. Leveraging ideas from the area of optimal transport, we initialize the systematic study of policy switching in offline RL. We establish fundamental properties and design a Net Actor-Critic algorithm for the proposed novel switching formulation. Numerical experiments demonstrate the efficiency of our approach on multiple robot control benchmarks of the Gymnasium and traffic light control from SUMO-RL.

📄 PDF Abstract BibTeX arXiv:2407.01837

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RL Reinforcement Learning (RL)

Evaluation-Time Policy Switching for Offline Reinforcement Learning

2025-03-15 · Natinael Solomon Neggatu, Jeremie Houssineau, Giovanni Montana

Offline reinforcement learning (RL) looks at learning how to optimally solve tasks using a fixed dataset of interactions from the environment. Many off-policy algorithms developed for online learning struggle in the offl…

Behavioural cloning Offline RL reinforcement-learning Reinforcement Learning+1

Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning

2024-12-25 · Yassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern 외

Offline safe reinforcement learning (OSRL) involves learning a decision-making policy to maximize rewards from a fixed batch of training data to satisfy pre-defined safety constraints. However, adapting to varying safety…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1

Fairness-Regularized Online Optimization with Switching Costs

2025-12-11 · Pengfei Li, Yuelin Han, Adam Wierman, Shaolei Ren arxiv

Fairness and action smoothness are two crucial considerations in many online optimization problems, but they have yet to be addressed simultaneously. In this paper, we study a new and challenging setting of fairness-regu…

A Benchmark for Low-Switching-Cost Reinforcement Learning

2021-12-13 · Shusheng Xu, Yancheng Liang, Yunfei Li, Simon Shaolei Du 외

A ubiquitous requirement in many practical reinforcement learning (RL) applications, including medical treatment, recommendation system, education and robotics, is that the deployed policy that actually interacts with th…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Deep Q-Learning with Low Switching Cost

2021-01-01 · Shusheng Xu, Simon Shaolei Du, Yi Wu

We initiate the study on deep reinforcement learning problems that require low switching cost, i.e., small number of policy switches during training. Such a requirement is ubiquitous in many applications, such as medica…

Atari Games Deep Reinforcement Learning Q-Learning Recommendation Systems+1