Offline Reinforcement Learning With Combinatorial Action Spaces

2024-10-28 · Matthew Landers, Taylor W. Killian, Hugo Barnes, Thomas Hartvigsen, Afsaneh Doryab

Reinforcement learning problems often involve large action spaces arising from the simultaneous execution of multiple sub-actions, resulting in combinatorial action spaces. Learning in combinatorial action spaces is difficult due to the exponential growth in action space size with the number of sub-actions and the dependencies among these sub-actions. In offline settings, this challenge is compounded by limited and suboptimal data. Current methods for offline learning in combinatorial spaces simplify the problem by assuming sub-action independence. We propose Branch Value Estimation (BVE), which effectively captures sub-action dependencies and scales to large combinatorial spaces by learning to evaluate only a small subset of actions at each timestep. Our experiments show that BVE outperforms state-of-the-art methods across a range of action space sizes.

📄 PDF Abstract BibTeX arXiv:2410.21151

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare

2023-05-02 · Shengpu Tang, Maggie Makar, Michael W. Sjoding, Finale Doshi-Velez 외

Many reinforcement learning (RL) applications have combinatorial action spaces, where each action is a composition of sub-actions. A standard RL approach ignores this inherent factorization structure, resulting in a pote…

Offline RL reinforcement-learning Reinforcement Learning (RL)

Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization

2026-01-07 · Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab arxiv

Reinforcement learning in discrete combinatorial action spaces requires searching over exponentially many joint actions to simultaneously select multiple sub-actions that form coherent combinations. Existing approaches e…

Reinforcement Learning Offline RL

Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling

2024-09-16 · Jesse van Remmerden, Zaharah Bukhsh, Yingqian Zhang

The Job Shop Scheduling Problem (JSSP) is a complex combinatorial optimization problem. While online Reinforcement Learning (RL) has shown promise by quickly finding acceptable solutions for JSSP, it faces key limitation…

Combinatorial Optimization counterfactual Job Shop Scheduling Q-Learning+4

An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces

2024-11-17 · Alex Beeson, David Ireland, Giovanni Montana

Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offli…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient Planning in Combinatorial Action Spaces with Applications to Cooperative Multi-Agent Reinforcement Learning

2023-02-08 · Volodymyr Tkachuk, Seyed Alireza Bakhtiari, Johannes Kirschner, Matej Jusup 외

A practical challenge in reinforcement learning are combinatorial action spaces that make planning computationally demanding. For example, in cooperative multi-agent reinforcement learning, a potentially large number of …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)