Towards Human Evaluation of Mutual Understanding in Human-Computer Spontaneous Conversation: An Empirical Study of Word Sense Disambiguation for Naturalistic Social Dialogs in American English
Current evaluation practices for social dialog systems, dedicated to human-computer spontaneous conversation, exclusively focus on the quality of system-generated surface text, but not human-verifiable aspects of mutual understanding between the systems and their interlocutors. This work proposes Word Sense Disambiguation (WSD) as an essential component of a valid and reliable human evaluation framework, whose long-term goal is to radically improve the usability of dialog systems in real-life human-computer collaboration. The practicality of this proposal is proved via experimentally investigating (1) the WordNet 3.0 sense inventory coverage of lexical meanings in spontaneous conversation between humans in American English, assumed as an upper bound of lexical diversity of human-computer communication, and (2) the effectiveness of state-of-the-art WSD models and pretrained transformer-based contextual embeddings on this type of data.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityvalidWord Sense DisambiguationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Mutual Theory of Mind for Human-AI Communication
New developments are enabling AI systems to perceive, recognize, and respond with social cues based on inferences made from humans' explicit or implicit behavioral and verbal cues. These AI systems, equipped with an equi…
Scene Image is Non-Mutually Exclusive - A Fuzzy Qualitative Scene Understanding
Ambiguity or uncertainty is a pervasive element of many real world decision making processes. Variation in decisions is a norm in this situation when the same problem is posed to different subjects. Psychological and met…
Binary ClassificationDecision MakingScene UnderstandingExplainable Interface for Human-Autonomy Teaming: A Survey
Nowadays, large-scale foundation models are being increasingly integrated into numerous safety-critical applications, including human-autonomy teaming (HAT) within transportation, medical, and defence domains. Consequent…
Explainable artificial intelligenceExplainable Artificial Intelligence (XAI)SurveyYou Impress Me: Dialogue Generation via Mutual Persona Perception
Despite the continuing efforts to improve the engagingness and consistency of chit-chat dialogue systems, the majority of current work simply focus on mimicking human-like responses, leaving understudied the aspects of m…
Dialogue GenerationMutualEyeContact: A conversation analysis tool with focus on eye contact
Eye contact between individuals is particularly important for understanding human behaviour. To further investigate the importance of eye contact in social interactions, portable eye tracking technology seems to be a nat…
Face Recognition