Corruption-robust exploration in episodic reinforcement learning
We initiate the study of multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system extending recent results for the special case of stochastic bandits. We provide a framework which modifies the aggressive exploration enjoyed by existing reinforcement learning approaches based on "optimism in the face of uncertainty", by complementing them with principles from "action elimination". Importantly, our framework circumvents the major challenges posed by naively applying action elimination in the RL setting, as formalized by a lower bound we demonstrate. Our framework yields efficient algorithms which (a) attain near-optimal regret in the absence of corruptions and (b) adapt to unknown levels corruption, enjoying regret guarantees which degrade gracefully in the total corruption encountered. To showcase the generality of our approach, we derive results for both tabular settings (where states and actions are finite) as well as linear-function-approximation settings (where the dynamics and rewards admit a linear underlying representation). Notably, our work provides the first sublinear regret guarantee which accommodates any deviation from purely i.i.d. transitions in the bandit-feedback model for episodic reinforcement learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-Armed Banditsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Improved Corruption Robust Algorithms for Episodic Reinforcement Learning
We study episodic reinforcement learning under unknown adversarial corruptions in both the rewards and the transition probabilities of the underlying system. We propose new algorithms which, compared to the existing resu…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Regioned Episodic Reinforcement Learning
Goal-oriented reinforcement learning algorithms are often good at exploration, not exploitation, while episodic algorithms excel at exploitation, not exploration. As a result, neither of these approaches alone can lead t…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward
Episodic count has been widely used to design a simple yet effective intrinsic motivation for reinforcement learning with a sparse reward. However, the use of episodic count in a high-dimensional state space as well as o…
Efficient Explorationreinforcement-learningELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
This paper proposes \emph{Episodic and Lifelong Exploration via Maximum ENTropy} (ELEMENT), a novel, multiscale, intrinsically motivated reinforcement learning (RL) framework that is able to explore environments without …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce a novel Episodic Multi-agent reinforcement learn…
Efficient ExplorationMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+3