Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act
Traditionally, Reinforcement Learning (RL) aims at deciding how to act optimally for an artificial agent. We argue that deciding when to act is equally important. As humans, we drift from default, instinctive or memorized behaviors to focused, thought-out behaviors when required by the situation. To enhance RL agents with this aptitude, we propose to augment the standard Markov Decision Process and make a new mode of action available: being lazy, which defers decision-making to a default policy. In addition, we penalize non-lazy actions in order to encourage minimal effort and have agents focus on critical decisions only. We name the resulting formalism lazy-MDPs. We study the theoretical properties of lazy-MDPs, expressing value functions and characterizing optimal solutions. Then we empirically demonstrate that policies learned in lazy-MDPs generally come with a form of interpretability: by construction, they show us the states where the agent takes control over the default policy. We deem those states and corresponding actions important since they explain the difference in performance between the default and the new, lazy policy. With suboptimal policies as default (pretrained or random), we observe that agents are able to get competitive performance in Atari games while only taking control in a limited subset of states.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesDecision Makingreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods
Current work in explainable reinforcement learning generally produces policies in the form of a decision tree over the state space. Such policies can be used for formal safety verification, agent behavior prediction, and…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Posterior Sampling for Reinforcement Learning Without Episodes
This is a brief technical note to clarify some of the issues with applying the application of the algorithm posterior sampling for reinforcement learning (PSRL) in environments without fixed episodes. In particular, this…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Provably Efficient Exploration in Quantum Reinforcement Learning with Logarithmic Worst-Case Regret
While quantum reinforcement learning (RL) has attracted a surge of attention recently, its theoretical understanding is limited. In particular, it remains elusive how to design provably efficient quantum RL algorithms th…
Efficient Explorationreinforcement-learningReinforcement Learning (RL)The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model
This paper investigates model robustness in reinforcement learning (RL) to reduce the sim-to-real gap in practice. We adopt the framework of distributionally robust Markov decision processes (RMDPs), aimed at learning a …
Reinforcement Learning (RL)Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling
Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. How…
Reinforcement Learning