Likelihood Reward Redistribution
In many practical reinforcement learning scenarios, feedback is provided only at the end of a long horizon, leading to sparse and delayed rewards. Existing reward redistribution methods typically assume that per-step rewards are independent, thus overlooking interdependencies among state--action pairs. In this paper, we propose a \emph{Likelihood Reward Redistribution} (LRR) framework that addresses this issue by modeling each per-step reward with a parametric probability distribution whose parameters depend on the state--action pair. By maximizing the likelihood of the observed episodic return via a leave-one-out (LOO) strategy that leverages the entire trajectory, our framework inherently introduces an uncertainty regularization term into the surrogate objective. Moreover, we show that the conventional mean squared error (MSE) loss for reward redistribution emerges as a special case of our likelihood framework when the uncertainty is fixed under the Gaussian distribution. When integrated with an off-policy algorithm such as Soft Actor-Critic, LRR yields dense and informative reward signals, resulting in superior sample efficiency and policy performance on Box-2d and MuJoCo benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoSimilar Papers 제목 키워드 기반
Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution
Reinforcement learning algorithms require many samples when solving complex hierarchical tasks with sparse and delayed rewards. For such complex tasks, the recently proposed RUDDER uses reward redistribution to leverage …
General Reinforcement LearningMinecraftMultiple Sequence AlignmentSafe ExplorationInterpretable Reward Redistribution in Reinforcement Learning: A Causal Approach
A major challenge in reinforcement learning is to determine which state-action pairs are responsible for future rewards that are delayed. Reward redistribution serves as a solution to re-assign credits for each time step…
reinforcement-learningReinforcement LearningRREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to …
Reinforcement LearningAgent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning
This paper considers multi-agent reinforcement learning (MARL) tasks where agents receive a shared global reward at the end of an episode. The delayed nature of this reward affects the ability of the agents to assess the…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)StarcraftReinforcement Learning from Bagged Reward
In Reinforcement Learning (RL), it is commonly assumed that an immediate reward signal is generated for each action taken by the agent, helping the agent maximize cumulative rewards to obtain the optimal policy. However,…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)