Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations

2018-01-31 · Xiaoqin Zhang, Huimin Ma

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert demonstrations. However, these methods are unstable and not suitable for actor-critic reinforcement learning algorithms. Also, some existing methods rely on the global optimum assumption, which is not true in most scenarios. In this paper, we employ expert demonstrations in a actor-critic reinforcement learning framework, and meanwhile ensure that the performance is not affected by the fact that expert demonstrations are not global optimal. We theoretically derive a method for computing policy gradients and value estimators with only expert demonstrations. Our method is theoretically plausible for actor-critic reinforcement learning algorithms that pretrains both policy and value functions. We apply our method to two of the typical actor-critic reinforcement learning algorithms, DDPG and ACER, and demonstrate with experiments that our method not only outperforms the RL algorithms without pretraining process, but also is more simulation efficient.

📄 PDF Abstract BibTeX arXiv:1801.10459

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Retrace Retrace is an off-policy Q-value estimation algorithm which has guaranteed convergence for a target and behaviour policy $\left(\pi, \beta\right)$. With off-policy rollout for…

Stochastic Dueling Network A Stochastic Dueling Network, or SDN, is an architecture for learning a value function $V$. The SDN learns both $V$ and $Q$ off-policy while maintaining consistency…

ACER ACER, or Actor Critic with Experience Replay, is an actor-critic deep reinforcement learning agent with [experience…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Actor-Critic Pretraining for Proximal Policy Optimization

2026-02-27 · Andreas Kernbach, Amr Elsheikh, Nicolas Grupp, René Nagel 외 arxiv

Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce…

Reinforcement Learning

Pretrain Soft Q-Learning with Imperfect Demonstrations

2019-05-09 · Xiaoqin Zhang, Yunfei Li, Huimin Ma, Xiong Luo

Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing rei…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations

2026-04-13 · Marcus Binder Nilsen, Julian Quick, Tuhfe Göçmen, Nikolay Dimitrov 외 arxiv

Reinforcement learning (RL) offers a promising approach for adaptive wind farm flow control, yet its practical deployment is hindered by slow training convergence and poor initial performance, factors that could translat…

Reinforcement Learning

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining

2023-10-12 · Licong Lin, Yu Bai, Song Mei

Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can make good decisions when prompted with int…

In-Context Reinforcement Learning reinforcement-learning Reinforcement Learning Thompson Sampling

Offline Actor-Critic Reinforcement Learning Scales to Large Models

2024-02-08 · Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth 외

We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outper…

continuous-control Continuous Control Offline RL reinforcement-learning+1