Empathy Through Multimodality in Conversational Interfaces
Agents represent one of the most emerging applications of Large Language Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal capabilities to navigate complex user environments. Conversational Health Agents (CHAs), a prime example of this, are redefining healthcare by offering nuanced support that transcends textual analysis to incorporate emotional intelligence. This paper introduces an LLM-based CHA engineered for rich, multimodal dialogue-especially in the realm of mental health support. It adeptly interprets and responds to users' emotional states by analyzing multimodal cues, thus delivering contextually aware and empathetically resonant verbal responses. Our implementation leverages the versatile openCHA framework, and our comprehensive evaluation involves neutral prompts expressed in diverse emotional tones: sadness, anger, and joy. We evaluate the consistency and repeatability of the planning capability of the proposed CHA. Furthermore, human evaluators critique the CHA's empathic delivery, with findings revealing a striking concordance between the CHA's outputs and evaluators' assessments. These results affirm the indispensable role of vocal (soon multimodal) emotion recognition in strengthening the empathetic connection built by CHAs, cementing their place at the forefront of interactive, compassionate digital health solutions.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotional IntelligenceEmotion RecognitionMultimodal Emotion RecognitionNavigateMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents
We introduce the concept of "empathic grounding" in conversational agents as an extension of Clark's conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the speak…
Emotional IntelligenceEmotion ClassificationHuman Interaction RecognitionLanguage Modelling+4From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agents
Empathy is a critical factor in fostering positive user experiences in conversational AI. While models can display empathy, it is often generic rather than tailored to specific tasks and contexts. In this work, we introd…
Empathy-driven Arabic Conversational Chatbot
Conversational models have witnessed a significant research interest in the last few years with the advancements in sequence generation models. A challenging aspect in developing human-like conversational models is enabl…
ChatbotDecoderImpact of Multimodal and Conversational AI on Learning Outcomes and Experience
Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, …
Multi-dimensional Evaluation of Empathetic Dialog Responses
Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Y…
Language ModelingLanguage Modelling