Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning
We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state's visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma's Revenge.
Code (1)
Tasks
Montezuma's Revengereinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Exploration with Limited Memory: Streaming Algorithms for Coin Tossing, Noisy Comparisons, and Multi-Armed Bandits
Consider the following abstract coin tossing problem: Given a set of $n$ coins with unknown biases, find the most biased coin using a minimal number of coin tosses. This is a common abstraction of various exploration pro…
Multi-Armed BanditsApplication of the Kelly Criterion to Prediction Markets
Betting markets are gaining in popularity. Mean beliefs generally differ from prices in prediction markets. Logarithmic utility is employed to study the risk and return adjustments to prices. Some consequences are descri…
PredictionCount Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limit…
Reinforcement LearningReinforcement Learning for Channel Coding: Learned Bit-Flipping Decoding
In this paper, we use reinforcement learning to find effective decoding strategies for binary linear codes. We start by reviewing several iterative decoding algorithms that involve a decision-making process at each step,…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)The Power of Adaptivity in Identifying Statistical Alternatives
This paper studies the trade-off between two different kinds of pure exploration: breadth versus depth. We focus on the most biased coin problem, asking how many total coin flips are required to identify a ``heavy'' coin…
Anomaly Detection