Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

2017-11-30 · Christopher Tegho, Paweł Budzianowski, Milica Gašić

In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinforcement learning methods are very promising but rely on epsilon-greedy exploration, thus subjecting the user to a random choice of action during learning. Alternative approaches such as Gaussian Process SARSA (GPSARSA) estimate uncertainties and are sample efficient, leading to better user experience, but on the expense of a greater computational complexity. This paper examines approaches to extract uncertainty estimates from deep Q-networks (DQN) in the context of dialogue management. We perform an extensive benchmark of deep Bayesian methods to extract uncertainty estimates, namely Bayes-By-Backprop, dropout, its concrete variation, bootstrapped ensemble and alpha-divergences, combining it with DQN algorithm.

📄 PDF Abstract BibTeX arXiv:1711.11486

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningDialogue ManagementEfficient ExplorationEfficient Neural NetworkManagement Reinforcement Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Sarsa Sarsa is an on-policy TD control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} + \gamma{Q}\left(S\_{t+1},…

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

2021-09-09 · EMNLP 2021 11 · Carel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck 외

The ability to identify and resolve uncertainty is crucial for the robustness of a dialogue system. Indeed, this has been confirmed empirically on systems that utilise Bayesian approaches to dialogue belief tracking. How…

What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation

2021-09-15 · Christian Geishauser, Songbo Hu, Hsien-Chin Lin, Nurul Lubis 외

The dialogue management component of a task-oriented dialogue system is typically optimised via reinforcement learning (RL). Optimisation via RL is highly susceptible to sample inefficiency and instability. The hierarchi…

Dialogue Management Management Reinforcement Learning (RL)

Clipping Loops for Sample-Efficient Dialogue Policy Optimisation

2021-06-01 · NAACL 2021 4 · Yen-chen Wu, Carl Edward Rasmussen

Training dialogue agents requires a large number of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue. In this paper, we propose loop-clipping policy optimisation (LCPO) …

Bayes-Adaptive Deep Model-Based Policy Optimisation

2020-10-29 · Tai Hoang, Ngo Anh Vien

We introduce a Bayesian (deep) model-based reinforcement learning method (RoMBRL) that can capture model uncertainty to achieve sample-efficient policy optimisation. We propose to formulate the model-based policy optimis…

model Model-based Reinforcement Learning

Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces

2018-02-11 · Gellért Weisz, Paweł Budzianowski, Pei-Hao Su, Milica Gašić

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a polic…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1