Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces
In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Spoken Dialogue SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
Dialogue policy learning for task-oriented dialogue systems has enjoyed great progress recently mostly through employing reinforcement learning methods. However, these approaches have become very sophisticated. It is tim…
DecoderMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONreinforcement-learning+3Ethical Challenges in Data-Driven Dialogue Systems
The use of dialogue systems as a medium for human-machine interaction is an increasingly prevalent paradigm. A growing number of dialogue systems use conversation strategies that are learned from large datasets. There ar…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)SimpleDS: A Simple Deep Reinforcement Learning Dialogue System
This paper presents 'SimpleDS', a simple and publicly available dialogue system trained with deep reinforcement learning. In contrast to previous reinforcement learning dialogue systems, this system avoids manual feature…
Deep Reinforcement LearningFeature Engineeringreinforcement-learningReinforcement Learning+1Offline Reinforcement Learning for Large Scale Language Action Spaces
Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corp…
Language ModelingLanguage ModellingOffline RLreinforcement-learning+2Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
In this work, we present a hybrid learning method for training task-oriented dialogue systems through online user interactions. Popular methods for learning task-oriented dialogues include applying reinforcement learning…
Dialogue State TrackingImitation Learningreinforcement-learningReinforcement Learning+2