Model-based Adversarial Meta-Reinforcement Learning
Meta-reinforcement learning (meta-RL) aims to learn from multiple training tasks the ability to adapt efficiently to unseen test tasks. Despite the success, existing meta-RL algorithms are known to be sensitive to the task distribution shift. When the test task distribution is different from the training task distribution, the performance may degrade significantly. To address this issue, this paper proposes Model-based Adversarial Meta-Reinforcement Learning (AdMRL), where we aim to minimize the worst-case sub-optimality gap -- the difference between the optimal return and the return that the algorithm achieves after adaptation -- across all tasks in a family of tasks, with a model-based approach. We propose a minimax objective and optimize it by alternating between learning the dynamics model on a fixed task and finding the adversarial task for the current model -- the task for which the policy induced by the model is maximally suboptimal. Assuming the family of tasks is parameterized, we derive a formula for the gradient of the suboptimality with respect to the task parameters via the implicit function theorem, and show how the gradient estimator can be efficiently implemented by the conjugate gradient method and a novel use of the REINFORCE estimator. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy in the worst-case performance over all tasks, the generalization power to out-of-distribution tasks, and in training and test time sample efficiency, over existing state-of-the-art meta-RL algorithms.
Code (1)
Tasks
continuous-controlContinuous ControlMeta Reinforcement Learningmodelreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Adversarial Training for Meta Reinforcement Learning
Meta Reinforcement Learning (MRL) enables an agent to learn from a limited number of past trajectories and extrapolate to a new task. In this paper, we attempt to improve the robustness of MRL. We build upon model-agnost…
Generative Adversarial NetworkMeta-LearningMeta Reinforcement Learningreinforcement-learning+2Meta-Cognition. An Inverse-Inverse Reinforcement Learning Approach for Cognitive Radars
This paper considers meta-cognitive radars in an adversarial setting. A cognitive radar optimally adapts its waveform (response) in response to maneuvers (probes) of a possibly adversarial moving target. A meta-cognitive…
reinforcement-learningReinforcement Learning (RL)Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
Learning from demonstrations has made great progress over the past few years. However, it is generally data hungry and task specific. In other words, it requires a large amount of data to train a decent model on a partic…
Decision MakingImitation LearningMeta-Learningreinforcement-learning+2Boosting Exploration in Multi-Task Reinforcement Learning using Adversarial Networks
Advancements in reinforcement learning (RL) have been remarkable in recent years. However, the limitations of traditional training methods have become increasingly evident, particularly in meta-RL settings where agents f…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Yet Meta Learning Can Adapt Fast, It Can Also Break Easily
Meta learning algorithms have been widely applied in many tasks for efficient learning, such as few-shot image classification and fast reinforcement learning. During meta training, the meta learner develops a common lear…
Few-Shot Image Classificationimage-classificationImage ClassificationMeta-Learning