SocialNLI: A Dialogue-Centric Social Inference Dataset
Making theory-of-mind inferences from human dialogue is a strong indicator of a model's underlying social abilities, which are fundamental for adept AI assistants. However, large language and reasoning models struggle to understand sophisticated social phenomena in transcript data, such as sarcasm and irony. To assess the weaknesses of current models and to identify their solutions, we introduce SocialNLI (SoNLI) -- the first social dialogue inference dataset. SoNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances like irony and sarcasm, paired with inferences, corresponding likelihood scores, and human-written explanations. We explore social inference analysis as a facet of theory-of-mind, and evaluate LLM and reasoning model theory-of-mind ability through multi-step counterfactual reasoning.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues
This paper addresses the problem of dialogue reasoning with contextualized commonsense inference. We curate CICERO, a dataset of dyadic conversations with five types of utterance-level reasoning-based inferences: cause, …
Answer GenerationAnswer SelectionSounding Board: A User-Centric and Content-Driven Social Chatbot
We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize. The system architecture consists of several components including spoken language processing, dialogue management, language generation, and…
ChatbotDialogue ManagementManagementText GenerationCommonsense-Focused Dialogues for Response Generation: An Empirical Study
Smooth and effective communication requires the ability to perform latent or explicit commonsense inference. Prior commonsense reasoning benchmarks (such as SocialIQA and CommonsenseQA) mainly focus on the discriminative…
Response GenerationText GenerationSocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Inferring social relations from dialogues is vital for building emotionally intelligent robots to interpret human language better and act accordingly. We model the social network as an And-or Graph, named SocAoG, for the…
Dialog Relation ExtractionRelationECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning
We introduce ECHo (Event Causality Inference via Human-Centric Reasoning), a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios. ECHo employs real-world human-centric deductive …
Diagnostic