IMPLI: Investigating NLI Models’ Performance on Figurative Language
Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction.
Code (1)
Tasks
Natural Language InferenceSimilar Papers 제목 키워드 기반
IMPLI: Investigating NLI Models' Performance on Figurative Language
Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences that require understanding of figurative l…
Natural Language InferenceSentenceInvestigating Robustness of Dialog Models to Popular Figurative Language Constructs
Humans often employ figurative language use in communication, including during interactions with dialog systems. Thus, it is important for real-world dialog systems to be able to handle popular figurative language constr…
Open-Domain DialogDoes the "most sinfully decadent cake ever" taste good? Answering Yes/No Questions from Figurative Contexts
Figurative language is commonplace in natural language, and while making communication memorable and creative, can be difficult to understand. In this work, we investigate the robustness of Question Answering (QA) models…
Question AnsweringLMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models
Figurative language is a challenge for language models since its interpretation is based on the use of words in a way that deviates from their conventional order and meaning. Yet, humans can easily understand and interpr…
SentenceIMPLI: Investigatng NLI Models' Performance on Figurative Language
Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative …
Natural Language Inference