Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have their own limitations. This paper proposes a novel latent action framework that treats the action spaces of an end-to-end dialog agent as latent variables and develops unsupervised methods in order to induce its own action space from the data. Comprehensive experiments are conducted examining both continuous and discrete action types and two different optimization methods based on stochastic variational inference. Results show that the proposed latent actions achieve superior empirical performance improvement over previous word-level policy gradient methods on both DealOrNoDeal and MultiWoz dialogs. Our detailed analysis also provides insights about various latent variable approaches for policy learning and can serve as a foundation for developing better latent actions in future research.
Code (3)
Tasks
Decision MakingDialogue GenerationDialogue ManagementGoal-Oriented Dialogue SystemsPolicy Gradient MethodsReinforcement LearningVariational InferenceSimilar Papers 제목 키워드 기반
Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
Dialogue policy learning for task-oriented dialogue systems has enjoyed great progress recently mostly through employing reinforcement learning methods. However, these approaches have become very sophisticated. It is tim…
DecoderMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONreinforcement-learning+3Deep Reinforcement Learning for Chatbots Using Clustered Actions and Human-Likeness Rewards
Training chatbots using the reinforcement learning paradigm is challenging due to high-dimensional states, infinite action spaces and the difficulty in specifying the reward function. We address such problems using clust…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+3Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management
Reinforcement learning (RL) has shown great promise for developing dialogue management (DM) agents that are non-myopic, conduct rich conversations, and maximize overall user satisfaction. Despite recent developments in R…
Dialogue ManagementDiversityManagementreinforcement-learning+2Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces
In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a polic…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Policy Networks with Two-Stage Training for Dialogue Systems
In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Re…
Deep Reinforcement LearningDialogue State TrackingGaussian ProcessesReinforcement Learning+2