Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors
Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our results suggest that IM causes noticeable change by increasing the initial rewards, but also altering the way the agent plays; and that GRM mitigated reward hacking in some scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review
The reinforcement learning research area contains a wide range of methods for solving the problems of intelligent agent control. Despite the progress that has been made, the task of creating a highly autonomous agent is …
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Intrinsic Motivation for Encouraging Synergistic Behavior
We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not in…
Reinforcement LearningExpanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop
Active inference is an ambitious theory that treats perception, inference and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognit…
Reinforcement LearningIntrinsic motivation as constrained entropy maximization
"Intrinsic motivation" refers to the capacity for intelligent systems to be motivated endogenously, i.e. by features of agential architecture itself rather than by learned associations between action and reward. This pap…
Intrinsic Motivation in Dynamical Control Systems
Biological systems often choose actions without an explicit reward signal, a phenomenon known as intrinsic motivation. The computational principles underlying this behavior remain poorly understood. In this study, we inv…