State-Aware Variational Thompson Sampling for Deep Q-Networks

2021-02-07 · Siddharth Aravindan, Wee Sun Lee

Thompson sampling is a well-known approach for balancing exploration and exploitation in reinforcement learning. It requires the posterior distribution of value-action functions to be maintained; this is generally intractable for tasks that have a high dimensional state-action space. We derive a variational Thompson sampling approximation for DQNs which uses a deep network whose parameters are perturbed by a learned variational noise distribution. We interpret the successful NoisyNets method \cite{fortunato2018noisy} as an approximation to the variational Thompson sampling method that we derive. Further, we propose State Aware Noisy Exploration (SANE) which seeks to improve on NoisyNets by allowing a non-uniform perturbation, where the amount of parameter perturbation is conditioned on the state of the agent. This is done with the help of an auxiliary perturbation module, whose output is state dependent and is learnt end to end with gradient descent. We hypothesize that such state-aware noisy exploration is particularly useful in problems where exploration in certain \textit{high risk} states may result in the agent failing badly. We demonstrate the effectiveness of the state-aware exploration method in the off-policy setting by augmenting DQNs with the auxiliary perturbation module.

📄 PDF Abstract BibTeX arXiv:2102.03719

Code (1)

nus-lid/sane 공식 구현 tf

Tasks

Thompson Sampling

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning

2025-01-16 · Siddharth Aravindan, Dixant Mittal, Wee Sun Lee

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the envi…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Thompson Sampling

Thompson Sampling via Local Uncertainty

2019-10-30 · ICML 2020 1 · Zhendong Wang, Mingyuan Zhou

Thompson sampling is an efficient algorithm for sequential decision making, which exploits the posterior uncertainty to address the exploration-exploitation dilemma. There has been significant recent interest in integrat…

Decision Making Multi-Armed Bandits Sequential Decision Making Thompson Sampling+1

VITS : Variational Inference Thompson Sampling for contextual bandits

2023-07-19 · Pierre Clavier, Tom Huix, Alain Durmus

In this paper, we introduce and analyze a variant of the Thompson sampling (TS) algorithm for contextual bandits. At each round, traditional TS requires samples from the current posterior distribution, which is usually i…

Multi-Armed Bandits Thompson Sampling Variational Inference

Thompson Sampling for High-Dimensional Sparse Linear Contextual Bandits

2022-11-11 · Sunrit Chakraborty, Saptarshi Roy, Ambuj Tewari

We consider the stochastic linear contextual bandit problem with high-dimensional features. We analyze the Thompson sampling algorithm using special classes of sparsity-inducing priors (e.g., spike-and-slab) to model the…

Multi-Armed Bandits Thompson Sampling Variational Inference Vocal Bursts Intensity Prediction

Thompson Sampling via Fine-Tuning of LLMs

2025-10-15 · Nicolas Menet, Aleksandar Terzić, Michael Hersche, Andreas Krause 외 arxiv

Bayesian optimization in large unstructured discrete spaces is often hindered by the computational cost of maximizing acquisition functions due to the absence of gradients. We propose a scalable alternative based on Thom…

Computational Efficiency Reinforcement Learning