Deep Q Learning from Dynamic Demonstration with Behavioral Cloning
Although Deep Reinforcement Learning (DRL) has proven its capability to learn optimal policies by directly interacting with simulation environments, scaling up a DRL model is difficult due to exploding computational complexity compared with a supervised learning model. This study proposes a novel approach integrating deep Q learning from dynamic demonstrations with a behavioral cloning model (DQfDD-BC), which includes a supervised learning technique of instructing a DRL model to enhance its performance. Specifically, the DQfDD-BC model leverages historical demonstrations to pre-train a supervised BC model and to consistently update it by using the generated dynamic demonstrations. Then the DQfDD-BC model manages the sample complexity by exploiting both the historical and generated demonstrations. An expert loss function is designed to compare actions generated by the DRL model with those obtained from the BC model to provide advantageous guidance for policy improvements. Experimental results in several OpenAI Gym environments show that the proposed approach adapts to different imperfection levels of demonstrations, and meanwhile, significantly accelerates the learning processes. As illustrated in an ablation study, the dynamic demonstration and expert loss mechanisms with the use of a BC model contribute to improving the learning convergence performance compared with the origin DQfD model.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningOpenAI GymQ-LearningSimilar Papers 제목 키워드 기반
Concurrent Training Improves the Performance of Behavioral Cloning from Observation
Learning from demonstration is widely used as an efficient way for robots to acquire new skills. However, it typically requires that demonstrations provide full access to the state and action sequences. In contrast, lear…
Imitation LearningDriving Through Ghosts: Behavioral Cloning with False Positives
Safe autonomous driving requires robust detection of other traffic participants. However, robust does not mean perfect, and safe systems typically minimize missed detections at the expense of a higher false positive rate…
Autonomous DrivingDisagreement-Regularized Imitation Learning
We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance …
continuous-controlContinuous ControlImitation LearningPosterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to impr…
Reinforcement LearningBehavioral Cloning from Noisy Demonstrations
We consider the problem of learning an optimal expert behavior policy given noisy demonstrations that contain observations from both optimal and non-optimal expert behaviors. Popular imitation learning algorithms, such a…
Imitation Learning