Double Q-learning

2010-12-01 · NeurIPS 2010 12 · Hado V. Hasselt

In some stochastic environments the well-known reinforcement learning algorithm Q-learning performs very poorly. This poor performance is caused by large overestimations of action values. These overestimations result from a positive bias that is introduced because Q-learning uses the maximum action value as an approximation for the maximum expected action value. We introduce an alternative way to approximate the maximum expected value for any set of random variables. The obtained double estimator method is shown to sometimes underestimate rather than overestimate the maximum expected value. We apply the double estimator to Q-learning to construct Double Q-learning, a new off-policy reinforcement learning algorithm. We show the new algorithm converges to the optimal policy and that it performs well in some settings in which Q-learning performs poorly due to its overestimation.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Protocol for Executing and Benchmarking Eight Computational Doublet-Detection Methods in Single-Cell RNA Sequencing Data Analysis

2021-01-21 · Nan Miles Xi, Jingyi Jessica Li

The existence of doublets is a key confounder in single-cell RNA sequencing (scRNA-seq) data analysis. Computational methods have been developed for detecting doublets from scRNA-seq data. We developed an R package Doubl…

Benchmarking

DoubleML -- An Object-Oriented Implementation of Double Machine Learning in Python

2021-04-07 · Philipp Bach, Victor Chernozhukov, Malte S. Kurz, Martin Spindler

DoubleML is an open-source Python library implementing the double machine learning framework of Chernozhukov et al. (2018) for a variety of causal models. It contains functionalities for valid statistical inference on ca…

BIG-bench Machine Learning valid

DoubleML -- An Object-Oriented Implementation of Double Machine Learning in R

2021-03-17 · Philipp Bach, Victor Chernozhukov, Malte S. Kurz, Martin Spindler 외

The R package DoubleML implements the double/debiased machine learning framework of Chernozhukov et al. (2018). It provides functionalities to estimate parameters in causal models based on machine learning methods. The d…

BIG-bench Machine Learning valid

Efficient Continuous Control with Double Actors and Regularized Critics

2021-06-06 · Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Xiu Li

How to obtain good value estimation is one of the key problems in Reinforcement Learning (RL). Current value estimation methods, such as DDPG and TD3, suffer from unnecessary over- or underestimation bias. In this paper,…

continuous-control Continuous Control Reinforcement Learning (RL)

Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Tasks

2022-03-22 · Haobo Jiang, Jin Xie, Jian Yang

Double Q-learning is a popular reinforcement learning algorithm in Markov decision process (MDP) problems. Clipped Double Q-learning, as an effective variant of Double Q-learning, employs the clipped double estimator to …

Q-Learning