Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
Artificial Intelligence has been used to help human complete difficult tasks in complicated environments by providing optimized strategies for decision-making or replacing the manual labour. In environments including multiple agents, such as football, the most common methods to train agents are Imitation Learning and Multi-Agent Reinforcement Learning (MARL). However, the agents trained by Imitation Learning cannot outperform the expert demonstrator, which makes humans hardly get new insights from the learnt policy. Besides, MARL is prone to the credit assignment problem. In environments with sparse reward signal, this method can be inefficient. The objective of our research is to create a novel reward shaping method by embedding contextual information in reward function to solve the aforementioned challenges. We demonstrate this in the Google Research Football (GRF) environment. We quantify the contextual information extracted from game state observation and use this quantification together with original sparse reward to create the shaped reward. The experiment results in the GRF environment prove that our reward shaping method is a useful addition to state-of-the-art MARL algorithms for training agents in environments with sparse reward signal.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingImitation LearningMulti-agent Reinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Grounding Natural Language Commands to StarCraft II Game States for Narration-Guided Reinforcement Learning
While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longsta…
Deep Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Starcraft+1Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning
Goal-conditioned reinforcement learning (RL) is an interesting extension of the traditional RL framework, where the dynamic environment and reward sparsity can cause conventional learning algorithms to fail. Reward shapi…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)BAMDP Shaping: a Unified Theoretical Framework for Intrinsic Motivation and Reward Shaping
Intrinsic motivation (IM) and reward shaping are common methods for guiding the exploration of reinforcement learning (RL) agents by adding pseudo-rewards. Designing these rewards is challenging, however, and they can co…
Reinforcement Learning (RL)Learning to Dynamically Select Between Reward Shaping Signals
Reinforcement learning (RL) algorithms often have the limitation of sample complexity. Previous research has shown that the reliance on large amounts of experience can be mitigated through the presence of additional feed…
Reinforcement Learning (RL)ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
Sparse rewards are a major bottleneck in multi-agent reinforcement learning (MARL), where simultaneous learning induces non-stationarity and makes reward design especially delicate. Reward shaping can accelerate learning…
Multi-agent Reinforcement Learning