A fine-grained comparison of pragmatic language understanding in humans and language models
Pragmatics and non-literal language understanding are essential to human communication, and present a long-standing challenge for artificial language models. We perform a fine-grained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials. We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks. We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor literal interpretations over heuristic-based distractors. We also find preliminary evidence that models and humans are sensitive to similar linguistic cues. Our results suggest that pragmatic behaviors can emerge in models without explicitly constructed representations of mental states. However, models tend to struggle with phenomena relying on social expectation violations.
Code (1)
Similar Papers 제목 키워드 기반
The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models
Current large language models (LLMs) have demonstrated emerging capabilities in social intelligence tasks, including implicature resolution (Sravanthi et al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), …
Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Humans use social context to specify preferences over behaviors, i.e. their reward functions. Yet, algorithms for inferring reward models from preference data do not take this social learning view into account. Inspired …
Pragmatically Informative Color Generation by Grounding Contextual Modifiers
Grounding language in contextual information is crucial for fine-grained natural language understanding. One important task that involves grounding contextual modifiers is color generation. Given a reference color "green…
Natural Language UnderstandingImproving Cooperation in Language Games with Bayesian Inference and the Cognitive Hierarchy
In two-player cooperative games, agents can play together effectively when they have accurate assumptions about how their teammate will behave, but may perform poorly when these assumptions are inaccurate. In language ga…
Bayesian InferencePUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities
LLMs have demonstrated remarkable capability for understanding semantics, but they often struggle with understanding pragmatics. To demonstrate this fact, we release a Pragmatics Understanding Benchmark (PUB) dataset con…
Instruction FollowingMultiple-choice