Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimate

2020-07-09 · Mirco Mutti, Lorenzo Pratissoli, Marcello Restelli

In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy? In this paper, we argue that the entropy of the state distribution induced by finite-horizon trajectories is a sensible target. Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a non-parametric, $k$-nearest neighbors estimate of the state distribution entropy. In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics. Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning a variety of meaningful reward-based tasks downstream.

📄 PDF Abstract BibTeX arXiv:2007.04640

Code (1)

muttimirco/mepol 공식 구현 pytorch

Tasks

continuous-control Continuous Control

A Policy Gradient Method for Task-Agnostic Exploration

2020-06-12 · ICML Workshop LifelongML 2020 7 · Mirco Mutti, Lorenzo Pratissoli, Marcello Restelli

continuous-control Continuous Control

On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift

2019-08-01 · Alekh Agarwal, Sham M. Kakade, Jason D. Lee, Gaurav Mahajan

Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces. However, little is known about even their most basic theoretical converge…

Policy Gradient Methods Reinforcement Learning

Self-Supervised Continuous Control without Policy Gradient

2021-01-01 · Hao Sun, Ziping Xu, Meng Fang, Yuhang Song 외

Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update. In this work, …

continuous-control Continuous Control MuJoCo Policy Gradient Methods+3

Meta-Reinforcement Learning of Structured Exploration Strategies

2018-02-20 · NeurIPS 2018 12 · Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel 외

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. Howe…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

How Log-Barrier Helps Exploration in Policy Optimization

2026-03-16 · Leonardo Cesani, Matteo Papini, Marcello Restelli arxiv

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the le…