Strangeness-driven Exploration in Multi-Agent Reinforcement Learning
Efficient exploration strategy is one of essential issues in cooperative multi-agent reinforcement learning (MARL) algorithms requiring complex coordination. In this study, we introduce a new exploration method with the strangeness that can be easily incorporated into any centralized training and decentralized execution (CTDE)-based MARL algorithms. The strangeness refers to the degree of unfamiliarity of the observations that an agent visits. In order to give the observation strangeness a global perspective, it is also augmented with the the degree of unfamiliarity of the visited entire state. The exploration bonus is obtained from the strangeness and the proposed exploration method is not much affected by stochastic transitions commonly observed in MARL tasks. To prevent a high exploration bonus from making the MARL training insensitive to extrinsic rewards, we also propose a separate action-value function trained by both extrinsic reward and exploration bonus, on which a behavioral policy to generate transitions is designed based. It makes the CTDE-based MARL algorithms more stable when they are used with an exploration method. Through a comparative evaluation in didactic examples and the StarCraft Multi-Agent Challenge, we show that the proposed exploration method achieves significant performance improvement in the CTDE-based MARL algorithms.
Code (1)
Tasks
Efficient ExplorationMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)StarcraftSimilar Papers 제목 키워드 기반
Target Strangeness: A Novel Conformal Prediction Difficulty Estimator
This paper introduces Target Strangeness, a novel difficulty estimator for conformal prediction (CP) that offers an alternative approach for normalizing prediction intervals (PIs). By assessing how atypical a prediction …
Conformal PredictionPredictionPrediction IntervalsregressionEpisodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce a novel Episodic Multi-agent reinforcement learn…
Efficient ExplorationMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+3Curiosity-Driven Multi-Agent Exploration with Mixed Objectives
Intrinsic rewards have been increasingly used to mitigate the sparse reward problem in single-agent reinforcement learning. These intrinsic rewards encourage the agent to look for novel experiences, guiding the agent to …
When Robots Rate Their Own Interactions: Engagement Validity and the Strangeness Failure
Human-robot interaction (HRI) evaluation relies almost exclusively on human-completed questionnaires, leaving the robot's perspective unexamined. We propose an \textit{inverted evaluation}, in which LLM-powered robots co…
Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
Hierarchical reinforcement learning (HRL) effectively improves agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how …
Hierarchical Reinforcement LearningMinecraftreinforcement-learningReinforcement Learning+1