A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs
Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.
Code (1)
Tasks
Natural Language InferenceNegationSimilar Papers 제목 키워드 기반
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP m…
Natural Language InferenceNegationKnow "No'' Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
While CLIP has significantly advanced multimodal understanding by bridging vision and language, the inability to grasp negation - such as failing to differentiate concepts like "parking" from "no parking" - poses substan…
Image GenerationImage SegmentationLanguage ModelingLanguage Modelling+6Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning
Language model probing is often used to test specific capabilities of models. However, conclusions from such studies may be limited when the probing benchmarks are small and lack statistical power. In this work, we intro…
In-Context LearningLanguage ModellingNegationSentenceTNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP
Vision-language models (VLMs), such as CLIP, have demonstrated strong performance across a range of downstream tasks. However, CLIP is still limited in negation understanding: the ability to recognize the absence or excl…
Image CaptioningImage GenerationImage RetrievalImage-text matching+6Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documents
Resolving the scope of a negation within a sentence is a challenging NLP task. The complexity of legal texts and the lack of annotated in-domain negation corpora pose challenges for state-of-the-art (SotA) models when pe…
NegationNegation Scope ResolutionSentence