Best Response Shaping
We investigate the challenge of multi-agent deep reinforcement learning in partially competitive environments, where traditional methods struggle to foster reciprocity-based cooperation. LOLA and POLA agents learn reciprocity-based cooperative policies by differentiation through a few look-ahead optimization steps of their opponent. However, there is a key limitation in these techniques. Because they consider a few optimization steps, a learning opponent that takes many steps to optimize its return may exploit them. In response, we introduce a novel approach, Best Response Shaping (BRS), which differentiates through an opponent approximating the best response, termed the "detective." To condition the detective on the agent's policy for complex games we propose a state-aware differentiable conditioning mechanism, facilitated by a question answering (QA) method that extracts a representation of the agent based on its behaviour on specific environment states. To empirically validate our method, we showcase its enhanced performance against a Monte Carlo Tree Search (MCTS) opponent, which serves as an approximation to the best response in the Coin Game. This work expands the applicability of multi-agent RL in partially competitive environments and provides a new pathway towards achieving improved social welfare in general sum games.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningQuestion AnsweringSimilar Papers 제목 키워드 기반
ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
Sparse rewards are a major bottleneck in multi-agent reinforcement learning (MARL), where simultaneous learning induces non-stationarity and makes reward design especially delicate. Reward shaping can accelerate learning…
Multi-agent Reinforcement LearningBRExIt: On Opponent Modelling in Expert Iteration
Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best-response oracles to impr…
Verifier-Induced Support Reshaping in On-Policy Optimization
We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifie…
Mathematical ReasoningReinforcement LearningInstruction FollowingPrivacy Protection via Joint Real and Reactive Load Shaping in Smart Grids
Frequent metering of electricity consumption is crucial for demand side management in smart grids. However, metered data can be processed fairly easily by employing well-established nonintrusive appliance load monitoring…
ManagementBeyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correc…
Reinforcement Learning