Weakly Coupled Deep Q-Networks

2023-10-28 · NeurIPS 2023 11

We propose weakly coupled deep Q-networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN "subagents", one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems, $3^{10}$ total actions, and a continuous state space.

📄 PDF Abstract BibTeX arXiv:2310.18803

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningQ-Learning

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Breaking the bonds of weak coupling: the dynamic causal modelling of oscillator amplitudes

2018-12-15

Models of coupled oscillators are useful in describing a wide variety of phenomena in physics, biology and economics. These models typically rest on the premise that the oscillators are weakly coupled, meaning that ampli…

Model Selection

Crosstalk Impacts on Homogeneous Weakly-Coupled Multicore Fiber Based IM/DD System

2018-11-13

We numerically discussed crosstalk impacts on homogeneous weakly-coupled multicore fiber based intensity modulation/direct-detection (IM/DD) systems taking into account mean crosstalk power fluctuation, walk-off between …

Decoupled Spatial Neural Attention for Weakly Supervised Semantic Segmentation

2018-03-07 · Tianyi Zhang, Guosheng Lin, Jianfei Cai, Tong Shen 외

Weakly supervised semantic segmentation receives much research attention since it alleviates the need to obtain a large amount of dense pixel-wise ground-truth annotations for the training images. Compared with other for…

Image Captioning Segmentation Semantic Segmentation Weakly supervised Semantic Segmentation+1

Deep reinforcement learning for weakly coupled MDP's with continuous actions

2024-06-03 · Francisco Robledo, Urtzi Ayesta, Konstantin Avrachenkov

This paper introduces the Lagrange Policy for Continuous Actions (LPCA), a reinforcement learning algorithm specifically designed for weakly coupled MDP problems with continuous action spaces. LPCA addresses the challeng…

Deep Reinforcement Learning global-optimization reinforcement-learning Reinforcement Learning

Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs

2026-06-12 · Tianhao Wu, Matthew Zurek, Weina Wang, Qiaomin Xie arxiv

We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model. Naive reduction to a tabular MDP leads to high complexit…