Ergodicity in reinforcement learning
In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e., the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Safe Exploration in Markov Decision Processes
In environments with uncertain dynamics exploration is necessary to learn how to perform well. Existing reinforcement learning algorithms provide strong exploration guarantees, but they tend to rely on an ergodicity assu…
Safe ExplorationReinforcement Learning with Function Approximation for Non-Markov Processes
We study reinforcement learning methods with linear function approximation under non-Markov state and cost processes. We first consider the policy evaluation method and show that the algorithm converges under suitable er…
Reinforcement LearningCuriosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent
Reinforcement learners are agents that learn to pick actions that lead to high reward. Ideally, the value of a reinforcement learner's policy approaches optimality--where the optimal informed policy is the one which maxi…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe ExplorationSpectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices
In this work a novel method to quantify spectral ergodicity for random matrices is presented. The new methodology combines approaches rooted in the metrics of Thirumalai-Mountain (TM) and Kullbach-Leibler (KL) divergence…
Deep Learning'Ergodicity Economics' is Pseudoscience
In a series of papers, Ole Peters and his collaborators claim that the 'conceptual basis of mainstream economic theory' is 'flawed' and that the approach they call 'ergodicity economics' gives 'reason to hope for a futur…