User Response and Sentiment Prediction for Automatic Dialogue Evaluation
Automatic evaluation is beneficial for open-domain dialog system development. However, standard word-overlap metrics (BLEU, ROUGE) do not correlate well with human judgements of open-domain dialog systems. In this work we propose to use the sentiment of the next user utterance for turn or dialog level evaluation. Specifically we propose three methods: one that predicts the next sentiment directly, and two others that predict the next user utterance using an utterance or a feedback generator model and then classify its sentiment. Experiments show our model outperforming existing automatic evaluation metrics on both written and spoken open-domain dialogue datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Dialogue EvaluationOpen-Domain DialogSimilar Papers 제목 키워드 기반
An Adversarial Approach to High-Quality, Sentiment-Controlled Neural Dialogue Generation
In this work, we propose a method for neural dialogue response generation that allows not only generating semantically reasonable responses according to the dialogue history, but also explicitly controlling the sentiment…
Dialogue GenerationResponse GenerationVocal Bursts Intensity PredictionSEPRG: Sentiment aware Emotion controlled Personalized Response Generation
Social chatbots have gained immense popularity, and their appeal lies not just in their capacity to respond to the diverse requests from users, but also in the ability to develop an emotional connection with users. To fu…
ChatbotDialogue GenerationResponse GenerationExploring Persona Sentiment Sensitivity in Personalized Dialogue Generation
Personalized dialogue systems have advanced considerably with the integration of user-specific personas into large language models (LLMs). However, while LLMs can effectively generate personalized responses, the influenc…
Dialogue GenerationSensitivityParalinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralinguistic information, such as sentiment, emo…
AttributeLanguage ModelingLanguage ModellingQuestion Answering+3Leveraging Implicit Feedback from Deployment Data in Dialogue
We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leve…