Probing Task-Oriented Dialogue Representation from Language Models
This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We approach the problem from two aspects: supervised classifier probe and unsupervised mutual information probe. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way. Meanwhile, we propose an unsupervised mutual information probe to evaluate the mutual dependence between a real clustering and a representation clustering. The goals of this empirical paper are to 1) investigate probing techniques, especially from the unsupervised mutual information aspect, 2) provide guidelines of pre-trained language model selection for the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringLanguage ModelingLanguage ModellingModel SelectionSimilar Papers 제목 키워드 기반
DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations
Language models pre-trained on general text have achieved impressive results in diverse fields. Yet, the distinct linguistic characteristics of task-oriented dialogues (TOD) compared to general text limit the practical u…
DiversitySlot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning
Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In…
Contrastive LearningIntent DetectionLanguage ModelingLanguage Modelling+4BootTOD: Bootstrap Task-oriented Dialogue Representations by Aligning Diverse Responses
Pre-trained language models have been successful in many scenarios. However, their usefulness in task-oriented dialogues is limited due to the intrinsic linguistic differences between general text and task-oriented dialo…
DiversityDialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Eva…
Domain AdaptationMulti-domain Dialogue State TrackingNatural Language UnderstandingFutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue
Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained…
Language ModelingLanguage Modelling