Directed Policy Gradient for Safe Reinforcement Learning with Human Advice

2018-08-13 · Hélène Plisnier, Denis Steckelmacher, Tim Brys, Diederik M. Roijers, Ann Nowé

Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from human feedback are unsafe: rewarding or punishing the agent a-posteriori cannot immediately prevent it from wrong-doing. In this paper, we extend Policy Gradient to make it robust to external directives, that would otherwise break the fundamentally on-policy nature of Policy Gradient. Our technique, Directed Policy Gradient (DPG), allows a teacher or backup policy to override the agent before it acts undesirably, while allowing the agent to leverage human advice or directives to learn faster. Our experiments demonstrate that DPG makes the agent learn much faster than reward-based approaches, while requiring an order of magnitude less advice.

📄 PDF Abstract BibTeX arXiv:1808.04096

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Methods 이 논문이 사용한 방법론

DPG Deterministic Policy Gradient, or DPG, is a policy gradient method for reinforcement learning. Instead of the policy function $\pi\left(.\mid{s}\right)$ being modeled as a…

Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning

2026-03-28 · Leixin Chang, Xinchen Yao, Ben Liu, Liangjing Yang 외 arxiv

On-policy reinforcement learning (RL) algorithms have demonstrated great potential in robotic control, where effective exploration is crucial for efficient and high-quality policy learning. However, how to encourage the …

Reinforcement Learning

Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

2026-04-21 · Qiang Liu, Adrienne Kline, Ermin Wei arxiv

Safe Reinforcement Learning from Human Feedback (Safe RLHF) has recently achieved empirical success in developing helpful and harmless large language models by decoupling human preferences regarding helpfulness and harml…

Reinforcement Learning

Safe Reinforcement Learning in a Simulated Robotic Arm

2023-11-28 · Luka Kovač, Igor Farkaš

Reinforcement learning (RL) agents need to explore their environments in order to learn optimal policies. In many environments and tasks, safety is of critical importance. The widespread use of simulators offers a number…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration+1

Vulnerability Analysis of Safe Reinforcement Learning via Inverse Constrained Reinforcement Learning

2026-02-18 · Jialiang Fan, Shixiong Jiang, Mengyu Liu, Fanxin Kong arxiv

Safe reinforcement learning (Safe RL) aims to ensure policy performance while satisfying safety constraints. However, most existing Safe RL methods assume benign environments, making them vulnerable to adversarial pertur…

Reinforcement Learning Adversarial Attack

Policy Gradient RL Algorithms as Directed Acyclic Graphs

2020-12-14 · Juan Jose Garau Luis

Meta Reinforcement Learning (RL) methods focus on automating the design of RL algorithms that generalize to a wide range of environments. The framework introduced in (Anonymous, 2020) addresses the problem by representin…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)