DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. To address the above issues, we integrate distributional RL and value function factorization methods by proposing a Distributional Value Function Factorization (DFAC) framework to generalize expected value function factorization methods to their DFAC variants. DFAC extends the individual utility functions from deterministic variables to random variables, and models the quantile function of the total return as a quantile mixture. To validate DFAC, we demonstrate DFAC's ability to factorize a simple two-step matrix game with stochastic rewards and perform experiments on all Super Hard tasks of StarCraft Multi-Agent Challenge, showing that DFAC is able to outperform expected value function factorization baselines.
Code (1)
Tasks
Multi-agent Reinforcement LearningQ-LearningSMACSMAC+StarcraftMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning
In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To addres…
Multi-agent Reinforcement Learningreinforcement-learningSMACStarcraftH3DFact: Heterogeneous 3D Integrated CIM for Factorization with Holographic Perceptual Representations
Disentangling attributes of various sensory signals is central to human-like perception and reasoning and a critical task for higher-order cognitive and neuro-symbolic AI systems. An elegant approach to represent this in…
PDFactor: Learning Tri-Perspective View Policy Diffusion Field for Multi-Task Robotic Manipulation
Robotic manipulation based on visual observations and natural language instructions is a long-standing challenge in robotics. Yet prevailing approaches model action distribution by adopting explicit or implicit repre…
DenoisingAn axiomatization of $Λ$-quantiles
We give an axiomatic foundation to $\Lambda$-quantiles, a family of generalized quantiles introduced by Frittelli et al. (2014) under the name of Lambda Value at Risk. Under mild assumptions, we show that these functiona…
Fully Parameterized Quantile Function for Distributional Reinforcement Learning
Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The k…
Atari GamesDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+1