Commonsense Knowledge Transfer for Pre-trained Language Models
Despite serving as the foundation models for a wide range of NLP benchmarks, pre-trained language models have shown limited capabilities of acquiring implicit commonsense knowledge from self-supervision alone, compared to learning linguistic and factual knowledge that appear more explicitly in the surface patterns in text. In this work, we introduce \textit{commonsense knowledge transfer}, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model. It first exploits general texts to form queries for extracting commonsense knowledge from the neural commonsense knowledge model and then refines the language model with two self-supervised objectives: \textit{commonsense mask infilling} and \textit{commonsense relation prediction}, which align human language with the underlying commonsense knowledge.Empirical results show that our approach consistently improves the model's performance on downstream tasks that require commonsense reasoning. Moreover, we find that the improvement is more significant in the few-shot setting. This suggests that our approach helps language models better transfer to downstream tasks without extensive supervision by injecting commonsense knowledge into their parameters.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingRelation PredictionTransfer LearningSimilar Papers 제목 키워드 기반
Commonsense Knowledge Transfer for Pre-trained Language Models
Despite serving as the foundation models for a wide range of NLP benchmarks, pre-trained language models have shown limited capabilities of acquiring implicit commonsense knowledge from self-supervision alone, compared t…
Language ModelingLanguage ModellingRelation PredictionTransfer LearningCLICKER: Attention-Based Cross-Lingual Commonsense Knowledge Transfer
Recent advances in cross-lingual commonsense reasoning (CSR) are facilitated by the development of multilingual pre-trained models (mPTMs). While mPTMs show the potential to encode commonsense knowledge for different lan…
Question AnsweringTransfer LearningDo Language Models Perform Generalizable Commonsense Inference?
Inspired by evidence that pretrained language models (LMs) encode commonsense knowledge, recent work has applied LMs to automatically populate commonsense knowledge graphs (CKGs). However, there is a lack of understandin…
Knowledge GraphsLanguage Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
Despite the success of generative pre-trained language models on a series of text generation tasks, they still suffer in cases where reasoning over underlying commonsense knowledge is required during generation. Existing…
Text GenerationImproving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning. However, they rely on expensive data annotation and time-consuming training. Thus, we focus on unsupe…
Natural Language InferenceTransfer LearningWinowhy