홈 › Papers

Training a Turn-level User Engagingness Predictor for Dialogues with Weak Supervision

The standard approach to evaluating dialogue engagingness is by measuring conversation turns per session (CTPS), which implies that the dialogue length is the main predictor of the user engagement with a dialogue system. The main limitation of CTPS is that it can be measured only at the session level, i.e., once the dialogue is already over. However, it is crucial for a dialogue system to continuously monitor user engagement throughout the dialogue session as well. Existing approaches to measuring turn-level engagingness require human annotations for training and lack interpretability of their scores. We pioneer an alternative approach, Remaining Depth as Engagingness Predictor (RDEP), which uses the remaining depth (RD) for each turn as the heuristic weak label for engagingness. RDEP does not require human annotations and also relates closely to CTPS, thus serving as a good learning proxy for this metric. In our experiments, we show that RDEP achieves the new state-of-the-art results on the fine-grained evaluation of dialog (FED) dataset (0.38 Spearman) and the Daily-Dialog dataset (0.62 Spearman).

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반