Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent

2024-02-05 · Yingru Li, Jiawei Xu, Lei Han, Zhi-Quan Luo

We propose HyperAgent, a reinforcement learning (RL) algorithm based on the hypermodel framework for exploration in RL. HyperAgent allows for the efficient incremental approximation of posteriors associated with an optimal action-value function ($Q^\star$) without the need for conjugacy and follows the greedy policies w.r.t. these approximate posterior samples. We demonstrate that HyperAgent offers robust performance in large-scale deep RL benchmarks. It can solve Deep Sea hard exploration problems with episodes that optimally scale with problem size and exhibits significant efficiency gains in the Atari suite. Implementing HyperAgent requires minimal code addition to well-established deep RL frameworks like DQN. We theoretically prove that, under tabular assumptions, HyperAgent achieves logarithmic per-step computational complexity while attaining sublinear regret, matching the best known randomized tabular RL algorithm.

📄 PDF Abstract BibTeX arXiv:2402.10228

Code (3)

szrlee/hyperagent 공식 구현 pytorch

szrlee/GPT-HyperAgent pytorch

szrlee/ensemble_plus_plus pytorch

Tasks

Atari GamesAtari Games 100kEfficient ExplorationLEMMAReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Scalable Nonparametric Sampling from Multimodal Posteriors with the Posterior Bootstrap

2019-02-08 · Edwin Fong, Simon Lyddon, Chris Holmes

Increasingly complex datasets pose a number of challenges for Bayesian inference. Conventional posterior sampling based on Markov chain Monte Carlo can be too computationally intensive, is serial in nature and mixes poor…

Bayesian Inference regression

Coordinated Exploration in Concurrent Reinforcement Learning

2018-02-05 · ICML 2018 7 · Maria Dimakopoulou, Benjamin Van Roy

We consider a team of reinforcement learning agents that concurrently learn to operate in a common environment. We identify three properties - adaptivity, commitment, and diversity - which are necessary for efficient coo…

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Theoretical guidelines for annealed Langevin dynamics in compositional simulation-based inference

2026-05-20 · Camille Touron, Gabriel V. Cardoso, Julyan Arbel, Pedro L. C. Rodrigues arxiv

Compositional score-based approaches to simulation-based inference (SBI) approximate the posterior over a shared parameter given $n$ independent observations by aggregating individually learned posterior scores: currentl…

Scalable Thompson Sampling via Optimal Transport

2019-02-19 · Ruiyi Zhang, Zheng Wen, Changyou Chen, Lawrence Carin

Thompson sampling (TS) is a class of algorithms for sequential decision-making, which requires maintaining a posterior distribution over a model. However, calculating exact posterior distributions is intractable for all …

Decision Making Sequential Decision Making Thompson Sampling

Deep importance sampling using tensor trains with application to a priori and a posteriori rare event estimation

2022-09-05 · Tiangang Cui, Sergey Dolgov, Robert Scheichl

We propose a deep importance sampling method that is suitable for estimating rare event probabilities in high-dimensional problems. We approximate the optimal importance distribution in a general importance sampling prob…

Bayesian Inference