Diversifying Neural Dialogue Generation via Negative Distillation
Generative dialogue models suffer badly from the generic response problem, limiting their applications to a few toy scenarios. Recently, an interesting approach, namely negative training, has been proposed to alleviate this problem by reminding the model not to generate high-frequency responses during training. However, its performance is hindered by two issues, ignoring low-frequency but generic responses and bringing low-frequency but meaningless responses. In this paper, we propose a novel negative training paradigm, called negative distillation, to keep the model away from the undesirable generic responses while avoiding the above problems. First, we introduce a negative teacher model that can produce query-wise generic responses, and then the student model is required to maximize the distance with multi-level negative knowledge. Empirical results show that our method outperforms previous negative training methods significantly.
Code (0)
등록된 구현이 없습니다.
Tasks
Dialogue GenerationSimilar Papers 제목 키워드 기반
Diversifying Neural Dialogue Generation via Negative Distillation
Generative dialogue models suffer from serious generic response problems, limiting their applications to a few toy scenarios. Recently, an interesting approach, namely negative training, has been proposed to alleviate t…
Dialogue GenerationHeterogeneous-Branch Collaborative Learning for Dialogue Generation
With the development of deep learning, advanced dialogue generation methods usually require a greater amount of computational resources. One promising approach to obtaining a high-performance and lightweight model is kno…
AttributeDialogue GenerationKnowledge DistillationTowards Efficiently Diversifying Dialogue Generation via Embedding Augmentation
Dialogue generation models face the challenge of producing generic and repetitive responses. Unlike previous augmentation methods that mostly focus on token manipulation and ignore the essential variety within a single s…
Dialogue GenerationDiversityDiversifying Dialogue Generation with Non-Conversational Text
Neural network-based sequence-to-sequence (seq2seq) models strongly suffer from the low-diversity problem when it comes to open-domain dialogue generation. As bland and generic utterances usually dominate the frequency d…
Dialogue GenerationDiversityTranslationDiversifying Dialog Generation via Adaptive Label Smoothing
Neural dialogue generation models trained with the one-hot target distribution suffer from the over-confidence issue, which leads to poor generation diversity as widely reported in the literature. Although existing appro…
DecoderDialogue GenerationDiversity