A Tailored Pre-Training Model for Task-Oriented Dialog Generation
The recent success of large pre-trained language models such as BERT and GPT-2 has suggested the effectiveness of incorporating language priors in downstream dialog generation tasks. However, the performance of pre-trained models on the dialog task is not as optimal as expected. In this paper, we propose a Pre-trained Role Alternating Language model (PRAL), designed specifically for task-oriented conversational systems. We adopted (Wu et al., 2019) that models two speakers separately. We also design several techniques, such as start position randomization, knowledge distillation, and history discount to improve pre-training performance. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. We test PRAL on three different downstream tasks. The results show that PRAL performs better or on par with state-of-the-art methods.
Code (1)
Tasks
Knowledge DistillationLanguage ModelingLanguage ModellingPositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation
Large pre-trained language generation models such as GPT-2 have demonstrated their effectiveness as language priors by reaching state-of-the-art results in various language generation tasks. However, the performance of p…
Knowledge DistillationLanguage ModelingLanguage ModellingPosition+1Towards Reliable and Empathetic Depression-Diagnosis-Oriented Chats
Chatbots can serve as a viable tool for preliminary depression diagnosis via interactive conversations with potential patients. Nevertheless, the blend of task-oriented and chit-chat in diagnosis-related dialogues necess…
Explicit State Tracking with Semi-Supervision for Neural Dialogue Generation
The task of dialogue generation aims to automatically provide responses given previous utterances. Tracking dialogue states is an important ingredient in dialogue generation for estimating users' intention. However, the …
DecoderDialogue GenerationDialogue State TrackingEmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems
The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues remain largel…
Emotion RecognitionEmotion Recognition in ConversationTask-Oriented Dialogue SystemsEmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems
The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues have been lar…
Emotion RecognitionTask-Oriented Dialogue Systems