Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Reward
Real-world decision-making problems are usually accompanied by delayed rewards, which affects the sample efficiency of Reinforcement Learning, especially in the extremely delayed case where the only feedback is the episodic reward obtained at the end of an episode. Episodic return decomposition is a promising way to deal with the episodic-reward setting. Several corresponding algorithms have shown remarkable effectiveness of the learned step-wise proxy rewards from return decomposition. However, these existing methods lack either attribution or representation capacity, leading to inefficient decomposition in the case of long-term episodes. In this paper, we propose a novel episodic return decomposition method called Diaster (Difference of implicitly assigned sub-trajectory reward). Diaster decomposes any episodic reward into credits of two divided sub-trajectories at any cut point, and the step-wise proxy rewards come from differences in expectation. We theoretically and empirically verify that the decomposed proxy reward function can guide the policy to be nearly optimal. Experimental results show that our method outperforms previous state-of-the-art methods in terms of both sample efficiency and performance.
Code (1)
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
MDP Planning as Policy Inference
We cast episodic Markov decision process (MDP) planning as Bayesian inference over policies. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its ex…
Bayesian InferenceLearning Long-Term Reward Redistribution via Randomized Return Decomposition
Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider…
Attributereinforcement-learningReinforcement LearningReinforcement Learning (RL)Segmenting Action-Value Functions Over Time-Scales in SARSA via TD($Δ$)
In numerous episodic reinforcement learning (RL) settings, SARSA-based methodologies are employed to enhance policies aimed at maximizing returns over long horizons. Conventional SARSA algorithms, however, have difficult…
Reinforcement Learning (RL)Extending Differential Temporal Difference Methods for Episodic Problems
Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the a…
Reinforcement LearningDisentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the qu…
continuous-controlContinuous ControlFuture predictionMuJoCo+2