Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
Reward decomposition is a critical problem in centralized training with decentralized execution~(CTDE) paradigm for multi-agent reinforcement learning. To take full advantage of global information, which exploits the states from all agents and the related environment for decomposing Q values into individual credits, we propose a general meta-learning-based Mixing Network with Meta Policy Gradient~(MNMPG) framework to distill the global hierarchy for delicate reward decomposition. The excitation signal for learning global hierarchy is deduced from the episode reward difference between before and after "exercise updates" through the utility network. Our method is generally applicable to the CTDE method using a monotonic mixing network. Experiments on the StarCraft II micromanagement benchmark demonstrate that our method just with a simple utility network is able to outperform the current state-of-the-art MARL algorithms on 4 of 5 super hard scenarios. Better performance can be further achieved when combined with a role-based utility network.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)StarcraftStarcraft IISimilar Papers 제목 키워드 기반
ProMP: Proximal Meta-Policy Search
Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-effic…
Meta-LearningMeta Reinforcement LearningReinforcement LearningHealth-Informed Policy Gradients for Multi-Agent Reinforcement Learning
This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the…
Multi-agent Reinforcement LearningPolicy Gradient Methodsreinforcement-learningReinforcement Learning+1On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy grad…
Guided Adaptive Credit Assignment for Sample Efficient Policy Optimization
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from sparse reward tasks, which leads to poor sample efficiency during tr…
Instruction FollowingPolicy Gradient MethodsAdaptive Pairwise Weights for Temporal Credit Assignment
How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely u…
Reinforcement Learning (RL)