An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
Large knowledge graphs have been shown to benefit zero-shot evaluation of downstream tasks, through continual pre-training of language models. Yet, little is known about how to optimally learn from this knowledge, and what is the impact of the resulting models on different task partitions. This paper studies the effect of model architectures, loss functions, and knowledge subsets on the generalization of zero-shot models across task partitions. Our experiments show that data size, model size, model architecture, and loss function all play an important role in the accuracy and generalizability of the models. Most of the improvement occurs on questions with short answers and dissimilar answer candidates, which corresponds to the characteristics of the data used for pre-training. These findings inform future work that uses self-supervision with large knowledge graphs in order to create generalizable commonsense reasoning agents.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge GraphsSimilar Papers 제목 키워드 기반
Commonsense Knowledge Transfer for Pre-trained Language Models
Despite serving as the foundation models for a wide range of NLP benchmarks, pre-trained language models have shown limited capabilities of acquiring implicit commonsense knowledge from self-supervision alone, compared t…
Language ModelingLanguage ModellingRelation PredictionTransfer LearningCommonsense Knowledge Transfer for Pre-trained Language Models
Despite serving as the foundation models for a wide range of NLP benchmarks, pre-trained language models have shown limited capabilities of acquiring implicit commonsense knowledge from self-supervision alone, compared t…
Language ModelingLanguage ModellingRelation PredictionTransfer LearningAn Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models, in zero-shot evaluation on various downstream language reasoning tasks. Sin…
DecoderKnowledge GraphsA Systematic Investigation of Commonsense Knowledge in Large Language Models
Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand the extent to which such models learn co…
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many convention…
graph constructionKnowledge Base ConstructionKnowledge Graphs