EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues
Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the human-annotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Towards a Multidimensional Evaluation Framework for Empathetic Conversational Systems
Empathetic Conversational Systems (ECS) are built to respond empathetically to the user's emotions and sentiments, regardless of the application domain. Current ECS studies evaluation approaches are restricted to offline…
BenchmarkingiEval: Interactive Evaluation Framework for Open-Domain Empathetic Chatbots
Building an empathetic chatbot is an important objective in dialog generation research, with evaluation being one of the most challenging parts. By empathy, we mean the ability to understand and relate to the speakers’ e…
ChatbotQuantitative Assessment of Intersectional Empathetic Bias and Understanding
A growing amount of literature critiques the current operationalizations of empathy based on loose definitions of the construct. Such definitions negatively affect dataset quality, model robustness, and evaluation reliab…
Multiple-choiceScoring with Large Language Models: A Study on Measuring Empathy of Responses in Dialogues
In recent years, Large Language Models (LLMs) have become increasingly more powerful in their ability to complete complex tasks. One such task in which LLMs are often employed is scoring, i.e., assigning a numerical valu…
feature selectionCoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The capacity of empathy is crucial to the success of open-domain dialog systems. Due to its nature of multi-dimensionality, there are various factors that relate to empathy expression, such as communication mechanism, di…
Empathetic Response GenerationOpen-Domain DialogResponse Generation