Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems
Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Common Structure Discovery in Collections of Bipartite Networks: Application to Pollination Systems
Bipartite networks are widely used to encode the ecological interactions. Being able to compare the organization of bipartite networks is a first step toward a better understanding of how environmental factors shape comm…
Link PredictionTransferable Dialogue Systems and User Simulators
One of the difficulties in training dialogue systems is the lack of training data. We explore the possibility of creating dialogue data through the interaction between a dialogue system and a user simulator. Our goal is …
Domain AdaptationTransfer LearningScriptWriter: Narrative-Guided Script Generation
It is appealing to have a system that generates a story or scripts automatically from a story-line, even though this is still out of our reach. In dialogue systems, it would also be useful to drive dialogues by a dialogu…
Script GenerationExploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking
Dialogue state tracking plays a crucial role in extracting information in task-oriented dialogue systems. However, preceding research are limited to textual modalities, primarily due to the shortage of authentic human au…
Dialogue State TrackingTask-Oriented Dialogue SystemsMore Informative Dialogue Generation via Multiple Knowledge Selection
Knowledge-grounded dialogue generation is a task of generating a fluent and informative response based on both dialogue context and a collection of external knowledge. There is a lot of noise in the knowledge pool, and a…
Dialogue Generation