Learning Efficient Dialogue Policy from Demonstrations through Shaping
Training a task-oriented dialogue agent with reinforcement learning is prohibitively expensive since it requires a large volume of interactions with users. Human demonstrations can be used to accelerate learning progress. However, how to effectively leverage demonstrations to learn dialogue policy remains less explored. In this paper, we present S{\^{}}2Agent that efficiently learns dialogue policy from demonstrations through policy shaping and reward shaping. We use an imitation model to distill knowledge from demonstrations, based on which policy shaping estimates feedback on how the agent should act in policy space. Reward shaping is then incorporated to bonus state-actions similar to demonstrations explicitly in value space encouraging better exploration. The effectiveness of the proposed S{\^{}}2Agentt is demonstrated in three dialogue domains and a challenging domain adaptation task with both user simulator evaluation and human evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationSimilar Papers 제목 키워드 기반
Policy Optimization with Demonstrations
Exploration remains a significant challenge to reinforcement learning methods, especially in environments where reward signals are sparse. Recent methods of learning from demonstrations have shown to be promising in…
Policy Gradient MethodsReinforcement LearningReinforcement Learning (RL)Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
Dialogue policy optimization often obtains feedback until task completion in task-oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only p…
Task-Oriented Dialogue SystemsDemonstration Actor Critic
We study the problem of \textit{Reinforcement learning from demonstrations (RLfD)}, where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverage…
Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems
Statistical spoken dialogue systems have the attractive property of being able to be optimised from data via interactions with real users. However in the reinforcement learning paradigm the dialogue manager (agent) often…
Reinforcement LearningSpoken Dialogue SystemsShaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
The potential benefits of model-free reinforcement learning to real robotics systems are limited by its uninformed exploration that leads to slow convergence, lack of data-efficiency, and unnecessary interactions with th…
Imitation Learningreinforcement-learningReinforcement Learning (RL)