EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World
Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users' internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents
Memory-augmented language agents are increasingly deployed in affective applications such as emotional support, where understanding and responding to users' latent emotional needs is critical. However, existing research …
From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment
Effective emotional support hinges on understanding users' emotions and needs to provide meaningful comfort during multi-turn interactions. Large Language Models (LLMs) show great potential for expressing empathy; howeve…
MindTailor: Personalized Emotional Support via Post History-Grounded Case Formulation and Collaborative Refinement
As mental health concerns continue to rise globally, social media has emerged as a vital space where individuals seek emotional support. While prior work on personalized emotional support has leveraged seekers' emotional…
The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs
When an AI assistant remembers that Sarah is a single mother working two jobs, does it interpret her stress differently than if she were a wealthy executive? As personalized AI systems increasingly incorporate long-term …
Emotional IntelligenceEvaluating the Impact of AI-Powered Audiovisual Personalization on Learner Emotion, Focus, and Learning Outcomes
Independent learners often struggle with sustaining focus and emotional regulation in unstructured or distracting settings. Although some rely on ambient aids such as music, ASMR, or visual backgrounds to support concent…