Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning

2022-01-12 · Baicen Xiao, Bhaskar Ramasubramanian, Radha Poovendran

This paper considers multi-agent reinforcement learning (MARL) tasks where agents receive a shared global reward at the end of an episode. The delayed nature of this reward affects the ability of the agents to assess the quality of their actions at intermediate time-steps. This paper focuses on developing methods to learn a temporal redistribution of the episodic reward to obtain a dense reward signal. Solving such MARL problems requires addressing two challenges: identifying (1) relative importance of states along the length of an episode (along time), and (2) relative importance of individual agents' states at any single time-step (among agents). In this paper, we introduce Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning (AREL) to address these two challenges. AREL uses attention mechanisms to characterize the influence of actions on state transitions along trajectories (temporal attention), and how each agent is affected by other agents at each time-step (agent attention). The redistributed rewards predicted by AREL are dense, and can be integrated with any given MARL algorithm. We evaluate AREL on challenging tasks from the Particle World environment and the StarCraft Multi-Agent Challenge. AREL results in higher rewards in Particle World, and improved win rates in StarCraft compared to three state-of-the-art reward redistribution methods. Our code is available at https://github.com/baicenxiao/AREL.

📄 PDF Abstract BibTeX arXiv:2201.04612

Code (1)

baicenxiao/arel 공식 구현 pytorch

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)Starcraft

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

2021-11-26 · ICLR 2022 4 · Zhizhou Ren, Ruihan Guo, Yuan Zhou, Jian Peng

Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider…

Attribute reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

$TAR^2$: Temporal-Agent Reward Redistribution for Optimal Policy Preservation in Multi-Agent Reinforcement Learning

2025-02-07 · Aditya Kapoor, Kale-ab Tessera, Mayank Baranwal, Harshad Khadilkar 외

In cooperative multi-agent reinforcement learning (MARL), learning effective policies is challenging when global rewards are sparse and delayed. This difficulty arises from the need to assign credit across both agents an…

Multi-agent Reinforcement Learning TAR

Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning

2024-12-15 · Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao 외

Reinforcement learning (RL) often encounters delayed and sparse feedback in real-world applications, even with only episodic rewards. Previous approaches have made some progress in reward redistribution for credit assign…

Decision Making Large Language Model reinforcement-learning Reinforcement Learning+1

Off-Beat Multi-Agent Reinforcement Learning

2022-05-27 · Wei Qiu, Weixun Wang, Rundong Wang, Bo An 외

We investigate model-free multi-agent reinforcement learning (MARL) in environments where off-beat actions are prevalent, i.e., all actions have pre-set execution durations. During execution durations, the environment ch…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Reinforcement Learning from Bagged Reward

2024-02-06 · Yuting Tang, Xin-Qiang Cai, Yao-Xiang Ding, Qiyu Wu 외

In Reinforcement Learning (RL), it is commonly assumed that an immediate reward signal is generated for each action taken by the agent, helping the agent maximize cumulative rewards to obtain the optimal policy. However,…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)