MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-Armed BanditsReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains
High-dimensional observations and complex real-world dynamics present major challenges in reinforcement learning for both function approximation and exploration. We address both of these challenges with two complementary…
Minecraftreinforcement-learningReinforcement LearningReinforcement Learning (RL)Boosting Reinforcement Learning and Planning with Demonstrations: A Survey
Although reinforcement learning has seen tremendous success recently, this kind of trial-and-error learning can be impractical or inefficient in complex environments. The use of demonstrations, on the other hand, enables…
Decision Makingreinforcement-learningReinforcement LearningSequential Decision Making+1Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge
The effectiveness of model training heavily relies on the quality of available training resources. However, budget constraints often impose limitations on data collection efforts. To tackle this challenge, we introduce c…
RLEMMO: Evolutionary Multimodal Optimization Assisted By Deep Reinforcement Learning
Solving multimodal optimization problems (MMOP) requires finding all optimal solutions, which is challenging in limited function evaluations. Although existing works strike the balance of exploration and exploitation thr…
Deep Reinforcement LearningDiversityreinforcement-learningReinforcement LearningBoosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States
Actor-critic (AC) algorithms are a class of model-free deep reinforcement learning algorithms, which have proven their efficacy in diverse domains, especially in solving continuous control problems. Improvement of explor…
continuous-controlContinuous ControlDeep Reinforcement LearningMuJoCo