Language models are not naysayers: An analysis of language models on negation benchmarks
Negation has been shown to be a major bottleneck for masked language models, such as BERT. However, whether this finding still holds for larger-sized auto-regressive language models (``LLMs'') has not been studied comprehensively. With the ever-increasing volume of research and applications of LLMs, we take a step back to evaluate the ability of current-generation LLMs to handle negation, a fundamental linguistic phenomenon that is central to language understanding. We evaluate different LLMs -- including the open-source GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks. Through systematic experimentation with varying model sizes and prompts, we show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
Code (1)
Tasks
NegationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we …
Natural Language InferenceNegationThunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding
Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scarce. We conduct a corpus-based analysis of Korean negation and show that …
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP m…
Natural Language InferenceNegationThunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding
Negation is a fundamental linguistic phenomenon that poses persistent challenges for Large Language Models (LLMs), particularly in tasks requiring deep semantic understanding. Existing benchmarks often treat negation as …
Multiple-choiceNatural Language InferenceNegationSentenceMaking Language Models Robust Against Negation
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised metho…
Natural Language UnderstandingNegationPredictionQuestion Answering+1