Clipping Loops for Sample-Efficient Dialogue Policy Optimisation

2021-06-01 · NAACL 2021 4 · Yen-chen Wu, Carl Edward Rasmussen

Training dialogue agents requires a large number of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue. In this paper, we propose loop-clipping policy optimisation (LCPO) to eliminate useless responses. LCPO consists of two stages: loop clipping and advantage clipping. In loop clipping, we clip off useless responses (called loops) from dialogue history (called trajectories). The clipped trajectories are more succinct than the original ones, and the estimation of state-value is more accurate. Second, in advantage clipping, we estimate and clip the advantages of useless responses and normal ones separately. The clipped advantage distinguish useless actions from others and reduce the probabilities of useless actions efficiently. In experiments on Cambridge Restaurant Dialogue System, LCPO uses only 260 training dialogues to achieve 80{\%} success rate, while PPO baseline requires 2160 dialogues. Besides, LCPO receives 3.7/5 scores in human evaluation where the agent interactively collects 100 real-user dialogues in training phase.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

2017-11-30 · Christopher Tegho, Paweł Budzianowski, Milica Gašić

In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinf…

Deep Reinforcement Learning Dialogue Management Efficient Exploration Efficient Neural Network+2

What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation

2021-09-15 · Christian Geishauser, Songbo Hu, Hsien-Chin Lin, Nurul Lubis 외

The dialogue management component of a task-oriented dialogue system is typically optimised via reinforcement learning (RL). Optimisation via RL is highly susceptible to sample inefficiency and instability. The hierarchi…

Dialogue Management Management Reinforcement Learning (RL)

Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces

2018-02-11 · Gellért Weisz, Paweł Budzianowski, Pei-Hao Su, Milica Gašić

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a polic…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL

2025-09-25 · Madeleine Dwyer, Adam Sobey, Adriane Chapman arxiv

Training large language models (LLMs) with reinforcement learning (RL) methods such as PPO and GRPO commonly relies on ratio clipping to stabilise updates. While effective at preventing instability, clipping discards inf…

Reinforcement Learning Mathematical Reasoning

Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

2021-09-09 · EMNLP 2021 11 · Carel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck 외

The ability to identify and resolve uncertainty is crucial for the robustness of a dialogue system. Indeed, this has been confirmed empirically on systems that utilise Bayesian approaches to dialogue belief tracking. How…