Solving Continuous Control via Q-learning
While there has been substantial success for solving continuous control with actor-critic methods, simpler critic-only methods such as Q-learning find limited application in the associated high-dimensional action spaces. However, most actor-critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.
Code (1)
Tasks
continuous-controlContinuous ControlMulti-agent Reinforcement LearningQ-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favoura…
continuous-controlContinuous ControlQ-LearningEvolving Spiking Neural Networks for Nonlinear Control Problems
Spiking Neural Networks are powerful computational modelling tools that have attracted much interest because of the bioinspired modelling of synaptic interactions between neurons. Most of the research employing spiking n…
Recent Developments in Machine Learning Methods for Stochastic Control and Games
Stochastic optimal control and games have a wide range of applications, from finance and economics to social sciences, robotics, and energy management. Many real-world applications involve complex models that have driven…
energy managementManagementContinuous-Time Fitted Value Iteration for Robust Policies
Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. Especially for continuous control, solving this differential equation and its extension the Hamilton-Ja…
continuous-controlContinuous ControlDeep Reinforcement LearningDeep Learning for Continuous-time Stochastic Control with Jumps
In this paper, we introduce a model-based deep-learning approach to solve finite-horizon continuous-time stochastic control problems with jumps. We iteratively train two neural networks: one to represent the optimal poli…
Deep Learning