Preventing Value Function Collapse in Ensemble Q-Learning by Maximizing Representation Diversity

2021-01-01 · Hassam Sheikh, Ladislau Boloni

The first deep RL algorithm, DQN, was limited by the overestimation bias of the learned Q-function. Subsequent algorithms proposed techniques to reduce this problem, without fully eliminating it. Recently, the Maxmin and Ensemble Q-learning algorithms used the different estimates provided by ensembles of learners to reduce the bias. Unfortunately, these learners can converge to the same point in the parametric or representation space, falling back to the classic single neural network DQN. In this paper, we describe a regularization technique to maximize diversity in the representation space in these algorithms. We propose and compare five regularization functions inspired from economics theory and consensus optimization. We show that the resulting approach significantly outperforms the Maxmin and Ensemble Q-learning algorithms as well as non-ensemble baselines.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityQ-Learning

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Preventing Value Function Collapse in Ensemble {Q}-Learning by Maximizing Representation Diversity

2020-06-24 · Hassam Ullah Sheikh, Ladislau Bölöni

The classic DQN algorithm is limited by the overestimation bias of the learned Q-function. Subsequent algorithms have proposed techniques to reduce this problem, without fully eliminating it. Recently, the Maxmin and Ens…

Diversity Q-Learning

TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse

2026-02-01 · Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh 외 arxiv

Despite scale driving substantial recent advancements in machine learning, reinforcement learning (RL) methods still primarily use small value functions. Naively scaling value functions -- including with a transformer ar…

Reinforcement Learning

Modulate Your Spectrum in Self-Supervised Learning

2023-05-26 · Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo 외

Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding an…

object-detection Object Detection Self-Supervised Learning

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning

2023-03-26 · Alex Beeson, Giovanni Montana

Offline reinforcement learning agents seek optimal policies from fixed data sets. With environmental interaction prohibited, agents face significant challenges in preventing errors in value estimates from compounding and…

Behavioural cloning Benchmarking reinforcement-learning

Anti-Collapse Loss for Deep Metric Learning Based on Coding Rate Metric

2024-07-03 · Xiruo Jiang, Yazhou Yao, Xili Dai, Fumin Shen 외

Deep metric learning (DML) aims to learn a discriminative high-dimensional embedding space for downstream tasks like classification, clustering, and retrieval. Prior literature predominantly focuses on pair-based and pro…

Diversity Metric Learning