Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
Winograd Schema Challenge (WSC) was proposed as an AI-hard problem in testing computers' intelligence on common sense representation and reasoning. This paper presents the new state-of-theart on WSC, achieving an accuracy of 71.1%. We demonstrate that the leading performance benefits from jointly modelling sentence structures, utilizing knowledge learned from cutting-edge pretraining models, and performing fine-tuning. We conduct detailed analyses, showing that fine-tuning is critical for achieving the performance, but it helps more on the simpler associative problems. Modelling sentence dependency structures, however, consistently helps on the harder non-associative subset of WSC. Analysis also shows that larger fine-tuning datasets yield better performances, suggesting the potential benefit of future work on annotating more Winograd schema sentences.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningSentenceSimilar Papers 제목 키워드 기반
Exploring Unsupervised Pretraining Objectives for Machine Translation
Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to se…
DecoderLanguage ModelingLanguage ModellingMachine Translation+3MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
Domain adaptive pretraining, i.e. the continued unsupervised pretraining of a language model on domain-specific text, improves the modelling of text for downstream tasks within the domain. Numerous real-world application…
Language ModelingLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+3Pretraining Reward-Free Representations for Data-Efficient Reinforcement Learning
Data efficiency poses a major challenge for deep reinforcement learning. We approach this issue from the perspective of self-supervised representation learning, leveraging reward-free exploratory data to pretrain encoder…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Document-level Neural Machine Translation Using Dependency RST Structure
Document-level machine translation (MT) extends the translation unit from the sentence to the whole document. Intuitively, discourse structure can be useful for document-level MT for its helpfulness in long-range depende…
DecoderDocument Level Machine TranslationMachine TranslationNMT+2Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval
Recent research demonstrates the effectiveness of using pretrained language models (PLM) to improve dense retrieval and multilingual dense retrieval. In this work, we present a simple but effective monolingual pretrainin…
Language ModellingPassage RetrievalRetrievalSentence+3