A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
Empowerment is an information-theoretic method that can be used to intrinsically motivate learning agents. It attempts to maximize an agent's control over the environment by encouraging visiting states with a large number of reachable next states. Empowered learning has been shown to lead to complex behaviors, without requiring an explicit reward signal. In this paper, we investigate the use of empowerment in the presence of an extrinsic reward signal. We hypothesize that empowerment can guide reinforcement learning (RL) agents to find good early behavioral solutions by encouraging highly empowered states. We propose a unified Bellman optimality principle for empowered reward maximization. Our empowered reward maximization approach generalizes both Bellman's optimality principle as well as recent information-theoretical extensions to it. We prove uniqueness of the empowered values and show convergence to the optimal solution. We then apply this idea to develop off-policy actor-critic RL algorithms which we validate in high-dimensional continuous robotics domains (MuJoCo). Our methods demonstrate improved initial and competitive final performance compared to model-free state-of-the-art techniques.
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Classical Policy Gradient: Preserving Bellman's Principle of Optimality
We propose a new objective function for finite-horizon episodic Markov decision processes that better captures Bellman's principle of optimality, and provide an expression for the gradient of the objective.
Direct and indirect reinforcement learning
Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how the…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1CertRL: Formalizing Convergence Proofs for Value and Policy Iteration in Coq
Reinforcement learning algorithms solve sequential decision-making problems in probabilistic environments by optimizing for long-term reward. The desire to use reinforcement learning in safety-critical settings inspires …
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1On Bellman's Optimality Principle for zs-POSGs
Many non-trivial sequential decision-making problems are efficiently solved by relying on Bellman's optimality principle, i.e., exploiting the fact that sub-problems are nested recursively within the original problem. He…
Decision MakingHeuristic SearchSequential Decision MakingBellman Gradient Iteration for Inverse Reinforcement Learning
This paper develops an inverse reinforcement learning algorithm aimed at recovering a reward function from the observed actions of an agent. We introduce a strategy to flexibly handle different types of actions with two …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)