Task-agnostic Exploration in Reinforcement Learning

2020-06-16 · NeurIPS 2020 12 · Xuezhou Zhang, Yuzhe ma, Adish Singla

Efficient exploration is one of the main challenges in reinforcement learning (RL). Most existing sample-efficient algorithms assume the existence of a single reward function during exploration. In many practical scenarios, however, there is not a single underlying reward function to guide the exploration, for instance, when an agent needs to learn many skills simultaneously, or multiple conflicting objectives need to be balanced. To address these challenges, we propose the \textit{task-agnostic RL} framework: In the exploration phase, the agent first collects trajectories by exploring the MDP without the guidance of a reward function. After exploration, it aims at finding near-optimal policies for $N$ tasks, given the collected trajectories augmented with \textit{sampled rewards} for each task. We present an efficient task-agnostic RL algorithm, \textsc{UCBZero}, that finds $\epsilon$-optimal policies for $N$ arbitrary tasks after at most $\tilde O(\log(N)H^5SA/\epsilon^2)$ exploration episodes. We also provide an $\Omega(\log (N)H^2SA/\epsilon^2)$ lower bound, showing that the $\log$ dependency on $N$ is unavoidable. Furthermore, we provide an $N$-independent sample complexity bound of \textsc{UCBZero} in the statistically easier setting when the ground truth reward functions are known.

📄 PDF Abstract BibTeX arXiv:2006.09497

Code (0)

등록된 구현이 없습니다.

Tasks

Efficient Explorationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Meta-Reinforcement Learning of Structured Exploration Strategies

2018-02-20 · NeurIPS 2018 12 · Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel 외

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. Howe…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Towards Principled Multi-Agent Task Agnostic Exploration

2025-02-12 · Riccardo Zamboni, Mirco Mutti, Marcello Restelli

In reinforcement learning, we typically refer to task-agnostic exploration when we aim to explore the environment without access to the task specification a priori. In a single-agent setting the problem has been extensiv…

Variable-Agnostic Causal Exploration for Reinforcement Learning

2024-07-17 · Minh Hoang Nguyen, Hung Le, Svetha Venkatesh

Modern reinforcement learning (RL) struggles to capture real-world cause-and-effect dynamics, leading to inefficient exploration due to extensive trial-and-error actions. While recent efforts to improve agent exploration…

Causal Discovery reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning

2023-04-14 · Gen Li, Yuling Yan, Yuxin Chen, Jianqing Fan

This paper studies reward-agnostic exploration in reinforcement learning (RL) -- a scenario where the learner is unware of the reward functions during the exploration stage -- and designs an algorithm that improves over …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Exploration Policies for Model-Agnostic Meta-Reinforcement Learning

2019-05-16 · ICML Workshop AMTL 2019 6 · Anonymous

Meta-Reinforcement learning approaches aim to develop learning procedures that can adapt quickly to a distribution of tasks with the help of a few examples. Developing efficient exploration strategies capable of finding …

Efficient Exploration Meta Reinforcement Learning reinforcement-learning Reinforcement Learning+1