Model-Based Imitation Learning Using Entropy Regularization of Model and Policy
Approaches based on generative adversarial networks for imitation learning are promising because they are sample efficient in terms of expert demonstrations. However, training a generator requires many interactions with the actual environment because model-free reinforcement learning is adopted to update a policy. To improve the sample efficiency using model-based reinforcement learning, we propose model-based Entropy-Regularized Imitation Learning (MB-ERIL) under the entropy-regularized Markov decision process to reduce the number of interactions with the actual environment. MB-ERIL uses two discriminators. A policy discriminator distinguishes the actions generated by a robot from expert ones, and a model discriminator distinguishes the counterfactual state transitions generated by the model from the actual ones. We derive structured discriminators so that the learning of the policy and the model is efficient. Computer simulations and real robot experiments show that MB-ERIL achieves a competitive performance and significantly improves the sample efficiency compared to baseline methods.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualImitation LearningmodelModel-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
In this paper, sample-aware policy entropy regularization is proposed to enhance the conventional policy entropy regularization for better exploration. Exploiting the sample distribution obtainable from the replay buffer…
DiversityEfficient Explorationreinforcement-learningReinforcement Learning (RL)Entropy annealing for policy mirror descent in continuous time and space
Entropy regularization has been widely used in policy optimization algorithms to enhance exploration and the robustness of the optimal control; however it also introduces an additional regularization bias. This work quan…
Policy Gradient MethodsMarginalized State Distribution Entropy Regularization in Policy Optimization
Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stoch…
continuous-controlContinuous ControlReinforcement LearningGeneralized Maximum Entropy Reinforcement Learning via Reward Shaping
Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy reg…
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Policy Optimization Reinforcement Learning with Entropy Regularization
Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy …
Continuous Controlreinforcement-learningReinforcement LearningReinforcement Learning (RL)