REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes

2024-01-16 · David Ireland, Giovanni Montana

Discrete-action reinforcement learning algorithms often falter in tasks with high-dimensional discrete action spaces due to the vast number of possible actions. A recent advancement leverages value-decomposition, a concept from multi-agent reinforcement learning, to tackle this challenge. This study delves deep into the effects of this value-decomposition, revealing that whilst it curtails the over-estimation bias inherent to Q-learning algorithms, it amplifies target variance. To counteract this, we present an ensemble of critics to mitigate target variance. Moreover, we introduce a regularisation loss that helps to mitigate the effects that exploratory actions in one dimension can have on the value of optimal actions in other dimensions. Our novel algorithm, REValueD, tested on discretised versions of the DeepMind Control Suite tasks, showcases superior performance, especially in the challenging humanoid and dog tasks. We further dissect the factors influencing REValueD's performance, evaluating the significance of the regularisation loss and the scalability of REValueD with increasing sub-actions per dimension.

📄 PDF Abstract BibTeX arXiv:2401.08850

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces

2024-11-17 · Alex Beeson, David Ireland, Giovanni Montana

Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offli…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Practical Quantum Advantage before Fault Tolerance via Quantum-Informed Machine Learning

2026-06-11 · Maida Wang, Xiao Xue, Minh Chung, Peter V. Coveney arxiv

Early quantum devices can deliver a practical advantage before fault tolerance. The role we identify is a statistical module within a classical scientific workflow: a compressed memory with a collective two-copy readout,…

Weather Forecasting

Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm

2025-10-28 · Clément Bénard arxiv

Tree ensembles have demonstrated state-of-the-art predictive performance across a wide range of problems involving tabular data. Nevertheless, the black-box nature of tree ensembles is a strong limitation, especially for…

Universal representation by Boltzmann machines with Regularised Axons

2023-10-22 · Przemysław R. Grzybowski, Antoni Jankiewicz, Eloy Piñol, David Cirauqui 외

It is widely known that Boltzmann machines are capable of representing arbitrary probability distributions over the values of their visible neurons, given enough hidden ones. However, sampling -- and thus training -- the…

Retrieval

Soft $Q(λ)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces

2026-04-15 · Pranav Mahajan, Ben Seymour arxiv

Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy. Despite its success,…

Reinforcement Learning