Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vessel

2022-02-23 · Haoran Zhang, Chenkun Yin, Yanxin Zhang, Shangtai Jin, Zhenxuan Li

In this paper, two novel practical methods of Reinforcement Learning from Demonstration (RLfD) are developed and applied to automatic berthing control systems for Unmanned Surface Vessel. A new expert data generation method, called Model Predictive Based Expert (MPBE) which combines Model Predictive Control and Deep Deterministic Policy Gradient, is developed to provide high quality supervision data for RLfD algorithms. A straightforward RLfD method, model predictive Deep Deterministic Policy Gradient (MP-DDPG), is firstly introduced by replacing the RL agent with MPBE to directly interact with the environment. Then distribution mismatch problem is analyzed for MP-DDPG, and two techniques that alleviate distribution mismatch are proposed. Furthermore, another novel RLfD algorithm based on the MP-DDPG, called Self-Guided Actor-Critic (SGAC) is present, which can effectively leverage MPBE by continuously querying it to generate high quality expert data online. The distribution mismatch problem leading to unstable learning process is addressed by SGAC in a DAgger manner. In addition, theoretical analysis is given to prove that SGAC algorithm can converge with guaranteed monotonic improvement. Simulation results verify the effectiveness of MP-DDPG and SGAC to accomplish the ship berthing control task, and show advantages of SGAC comparing with other typical reinforcement learning algorithms and MP-DDPG.

📄 PDF Abstract BibTeX arXiv:2202.11325

Code (0)

등록된 구현이 없습니다.

Tasks

Model Predictive Controlreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization

2026-05-01 · Yue Mao, Shicheng Liu, Siyuan Xu, Minghui Zhu arxiv

Inverse reinforcement learning (IRL) learns a reward function and a corresponding policy that best fit the demonstration data of an expert. However, in the current IRL setting, the learner is isolated from the expert and…

Reinforcement Learning

Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

2018-06-22 · Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu 외

Humans are able to understand and perform complex tasks by strategically structuring the tasks into incremental steps or subgoals. For a robot attempting to learn to perform a sequential task with critical subgoal states…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback

2021-04-14 · Jie Huang, Rongshun Juan, Randy Gomez, Keisuke Nakamura 외

Deep reinforcement learning (DRL) has achieved great successes in many simulated tasks. The sample inefficiency problem makes applying traditional DRL methods to real-world robots a great challenge. Generative Adversaria…

Deep Reinforcement Learning Imitation Learning reinforcement-learning Reinforcement Learning+1

Multi-Agent Generative Adversarial Interactive Self-Imitation Learning for AUV Formation Control and Obstacle Avoidance

2024-01-21 · Zheng Fang, Tianhao Chen, Dong Jiang, Zheng Zhang 외

Multiple autonomous underwater vehicles (multi-AUV) can cooperatively accomplish tasks that a single AUV cannot complete. Recently, multi-agent reinforcement learning has been introduced to control of multi-AUV. However,…

Imitation Learning Multi-agent Reinforcement Learning

Automatic Curricula via Expert Demonstrations

2021-06-16 · Siyu Dai, Andreas Hofmann, Brian Williams

We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipula…

Imitation Learning Reinforcement Learning (RL)