Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorporate uncertainty via distributional output have been integrated with multi-agent reinforcement learning (MARL) methods, achieving state-of-the-art performance. However, distributional MARL algorithms still rely on the traditional $\epsilon$-greedy, which does not take cooperative strategy into account. In this paper, we present a risk-based exploration that leads to collaboratively optimistic behavior by shifting the sampling region of distribution. Initially, we take expectations from the upper quantiles of state-action values for exploration, which are optimistic actions, and gradually shift the sampling region of quantiles to the full distribution for exploitation. By ensuring that each agent is exposed to the same level of risk, we can force them to take cooperatively optimistic actions. Our method shows remarkable performance in multi-agent settings requiring cooperative exploration based on quantile regression appropriately controlling the level of risk.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement LearningMulti-agent Reinforcement Learningquantile regressionreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning
Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this work, we propose an exploration method that effectively encourages cooperative exploration based on the idea of seq…
Efficient ExplorationMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning
The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, due to the representational limitations of traditional monotonic value deco…
Multi-agent Reinforcement LearningStrategically Efficient Exploration in Competitive Multi-agent Reinforcement Learning
High sample complexity remains a barrier to the application of reinforcement learning (RL), particularly in multi-agent systems. A large body of work has demonstrated that exploration mechanisms based on the principle of…
Efficient ExplorationMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning
Risk sensitivity has become a central theme in reinforcement learning (RL), where convex risk measures and robust formulations provide principled ways to model preferences beyond expected return. Recent extensions to mul…
Multi-agent Reinforcement LearningSocial learning via actions in bandit environments
I study a game of strategic exploration with private payoffs and public actions in a Bayesian bandit setting. In particular, I look at cascade equilibria, in which agents switch over time from the risky action to the ris…