Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

2021-01-15 · ICLR 2021 1 · Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross

Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.

📄 PDF Abstract BibTeX arXiv:2101.05982

Code (6)

watchernyu/REDQ 공식 구현 pytorch

BY571/Randomized-Ensembled-Double-Q-learning-REDQ- pytorch

LucasAlegre/sac-plus pytorch

facebookresearch/rl/tree/main/examples/redq/redq.py jax

trackmania-rl/tmrl pytorch

ustcmike/adaeq_neurips21 pytorch

Tasks

MuJoCoQ-Learning

Methods 이 논문이 사용한 방법론

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dropout Q-Functions for Doubly Efficient Reinforcement Learning

2021-10-05 · ICLR 2022 4 · Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi 외

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made …

Computational Efficiency Q-Learning reinforcement-learning Reinforcement Learning+1

Sequential Least-Squares Estimators with Fast Randomized Sketching for Linear Statistical Models

2025-09-08 · Guan-Yu Chen, Dong-Yue Xie, Xi Yang arxiv

We propose a novel randomized framework for the estimation problem of large-scale linear statistical models, namely Sequential Least-Squares Estimators with Fast Randomized Sketching (SLSE-FRS), which integrates Sketch-a…

Double Randomized Underdamped Langevin with Dimension-Independent Convergence Guarantee

2023-09-21 · NeurIPS 2023 11

This paper focuses on the high-dimensional sampling of log-concave distributions with composite structures: $p^*(\mathrm{d}x)\propto \exp(-g(x)-f(x))\mathrm{d}x$. We develop a double randomization technique, which leads …

REALITrees: Rashomon Ensemble Active Learning for Interpretable Trees

2026-03-24 · Simon D. Nguyen, Hayden McTavish, Kentaro Hoffman, Cynthia Rudin 외 arxiv

Active learning reduces labeling costs by selecting samples that maximize information gain. A dominant framework, Query-by-Committee (QBC), typically relies on perturbation-based diversity by inducing model disagreement …

Active Learning

Nonconvex Robust High-Order Tensor Completion Using Randomized Low-Rank Approximation

2023-05-19 · Wenjin Qin, Hailin Wang, Feng Zhang, Weijun Ma 외

Within the tensor singular value decomposition (T-SVD) framework, existing robust low-rank tensor completion approaches have made great achievements in various areas of science and engineering. Nevertheless, these method…

Computational Efficiency