Making Language Models Robust Against Negation
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised method to make language models more robust against negation. We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks. Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingNegationPredictionQuestion AnsweringSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Paraphrasing in Affirmative Terms Improves Negation Understanding
Negation is a common linguistic phenomenon. Yet language models face challenges with negation in many natural language understanding tasks such as question answering and natural language inference. In this paper, we expe…
Natural Language InferenceNatural Language UnderstandingNegationQuestion AnsweringLanguage models are not naysayers: An analysis of language models on negation benchmarks
Negation has been shown to be a major bottleneck for masked language models, such as BERT. However, whether this finding still holds for larger-sized auto-regressive language models (``LLMs'') has not been studied compre…
NegationAn Analysis of Natural Language Inference Benchmarks through the Lens of Negation
Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we …
Natural Language InferenceNegationNegation: A Pink Elephant in the Large Language Models' Room?
Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language models (LLMs) and remain underexplored. …
Language ModelingLanguage ModellingLogical ReasoningNatural Language Inference+2Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP m…
Natural Language InferenceNegation