When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
Generative adversarial imitation learning (GAIL) is a popular inverse reinforcement learning approach for jointly optimizing policy and reward from expert trajectories. A primary question about GAIL is whether applying a certain policy gradient algorithm to GAIL attains a global minimizer (i.e., yields the expert policy), for which existing understanding is very limited. Such global convergence has been shown only for the linear (or linear-type) MDP and linear (or linearizable) reward. In this paper, we study GAIL under general MDP and for nonlinear reward function classes (as long as the objective function is strongly concave with respect to the reward parameter). We characterize the global convergence with a sublinear rate for a broad range of commonly used policy gradient algorithms, all of which are implemented in an alternating manner with stochastic gradient ascent for reward update, including projected policy gradient (PPG)-GAIL, Frank-Wolfe policy gradient (FWPG)-GAIL, trust region policy optimization (TRPO)-GAIL and natural policy gradient (NPG)-GAIL. This is the first systematic theoretical study of GAIL for global convergence.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Risk-Sensitive Generative Adversarial Imitation Learning
We study risk-sensitive imitation learning where the agent's goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the…
Imitation LearningMuJoCoHybrid Adversarial Imitation Learning
Extrapolating beyond-demonstrator (BD) performance through the imitation learning (IL) algorithm aims to learn from and outperform the demonstrator. Most existing BDIL algorithms are performed in two stages by first infe…
Imitation LearningReinforcement Learning (RL)No Need for Interactions: Robust Model-Based Imitation Learning using Neural ODE
Interactions with either environments or expert policies during training are needed for most of the current imitation learning (IL) algorithms. For IL problems with no interactions, a typical approach is Behavior Cloning…
Imitation LearningMuJoCoOn the Benefits of Inducing Local Lipschitzness for Robust Generative Adversarial Imitation Learning
We explore methodologies to improve the robustness of generative adversarial imitation learning (GAIL) algorithms to observation noise. Towards this objective, we study the effect of local Lipschitzness of the discrimina…
Imitation LearningMuJoCoLearning to Generate Chairs with Generative Adversarial Nets
Generative adversarial networks (GANs) has gained tremendous popularity lately due to an ability to reinforce quality of its predictive model with generated objects and the quality of the generative model with and superv…