Thompson sampling for improved exploration in GFlowNets
Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. Unlike other algorithms for hierarchical sampling that optimize a variational bound, GFlowNet algorithms can stably run off-policy, which can be advantageous for discovering modes of the target distribution. Despite this flexibility in the choice of behaviour policy, the optimal way of efficiently selecting trajectories for training has not yet been systematically explored. In this paper, we view the choice of trajectories for training as an active learning problem and approach it using Bayesian techniques inspired by methods for multi-armed bandits. The proposed algorithm, Thompson sampling GFlowNets (TS-GFN), maintains an approximate posterior distribution over policies and samples trajectories from this posterior for training. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningDecision MakingMulti-Armed BanditsSequential Decision MakingThompson SamplingVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Thompson Sampling and Approximate Inference
We study the effects of approximate inference on the performance of Thompson sampling in the $k$-armed bandit problems. Thompson sampling is a successful algorithm for online decision-making but requires posterior infere…
Decision MakingThompson SamplingThompson Sampling with Approximate Inference
We study the effects of approximate inference on the performance of Thompson sampling in the $k$-armed bandit problems. Thompson sampling is a successful algorithm for online decision-making but requires posterior infere…
Decision MakingThompson SamplingDynamic Prior Thompson Sampling for Cold-Start Exploration in Recommender Systems
Cold-start exploration is a core challenge in large-scale recommender systems: new or data-sparse items must receive traffic to estimate value, but over-exploration harms users and wastes impressions. In practice, Thomps…
Exploring Multiple High-Scoring Subspaces in Generative Flow Networks
As a probabilistic sampling framework, Generative Flow Networks (GFlowNets) show strong potential for constructing complex combinatorial objects through the sequential composition of elementary components. However, exist…
Thompson Sampling via Local Uncertainty
Thompson sampling is an efficient algorithm for sequential decision making, which exploits the posterior uncertainty to address the exploration-exploitation dilemma. There has been significant recent interest in integrat…
Decision MakingMulti-Armed BanditsSequential Decision MakingThompson Sampling+1