Contrastive Self-Supervised Learning for Commonsense Reasoning
We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called "trigger" words, which are responsible for flipping the answer in pronoun disambiguation. We achieve such commonsense reasoning by constructing pair-wise contrastive auxiliary predictions. To this end, we leverage a mutual exclusive loss regularized by a contrastive margin. Our architecture is based on the recently introduced transformer networks, BERT, that exhibits strong performance on many NLP benchmarks. Empirical results show that our method alleviates the limitation of current supervised approaches for commonsense reasoning. This study opens up avenues for exploiting inexpensive self-supervision to achieve performance gain in commonsense reasoning tasks.
Code (3)
Tasks
Self-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
elBERto: Self-supervised Commonsense Learning for Question Answering
Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoni…
Question AnsweringRepresentation LearningRetrievalAttention-based Contrastive Learning for Winograd Schemas
Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective. This paper investigates whether contrastive lear…
Contrastive LearningSelf-Supervised LearningTowards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models
Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge…
Language ModelingLanguage ModellingSelf-Supervised LearningStr-GCL: Structural Commonsense Driven Graph Contrastive Learning
Graph Contrastive Learning (GCL) is a widely adopted approach in self-supervised graph representation learning, applying contrastive objectives to produce effective representations. However, current GCL methods primarily…
Graph Representation LearningContrastive LearningPre-training Text-to-Text Transformers to Write and Reason with Concepts
Pretrained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks that require a syntactic and semantic understanding of the text. However, c…
Common Sense ReasoningLanguage ModellingNatural Language UnderstandingTransfer Learning