Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75--80% on the English dataset and 58--69% in Korean, and models predict an Unlikely relationship in 10--25% of responses in both languages. Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases. In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sorting in Networks: Adversity and Structure
People choose friendships with people similar to themselves, i.e. they sort by resemblence. Economic studies have shown when sorting is optimal and constitute an equilibrium, however, this presumes lack of beneficial spi…
Content Creation with Spillovers: An Incentive Design Approach
The rise of AI amplifies the economic phenomenon of \emph{positive spillovers}: when creators contribute content that can be reused and adapted by LLMs, one creator's effort may improve the content quality of others thro…
A Neural Attention Model for Adaptive Learning of Social Friends' Preferences
Social-based recommendation systems exploit the selections of friends to combat the data sparsity on user preferences, and improve the recommendation accuracy of the collaborative filtering strategy. The main challenge i…
Collaborative FilteringRecommendation SystemsThe happiness paradox: your friends are happier than you
Most individuals in social networks experience a so-called Friendship Paradox: they are less popular than their friends on average. This effect may explain recent findings that widespread social network media use leads t…
Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory
As large language models (LLMs) are increasingly applied for emotional companionship, evaluating their behavior and capabilities in intimate relationships has become a pressing issue. However, existing assessments primar…