SenseBERT: Driving Some Sense into BERT
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingNatural Language InferenceNatural Language UnderstandingWord Sense DisambiguationSimilar Papers 제목 키워드 기반
KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding
Contextualized entity representations learned by state-of-the-art transformer-based language models (TLMs) like BERT, GPT, T5, etc., leverage the attention mechanism to learn the data context from training data corpus. H…
Graph EmbeddingKnowledge Graph EmbeddingKnowledge Graph EmbeddingsKnowledge Graphs+2Representation Learning of Image Schema
Image schema is a recurrent pattern of reasoning where one entity is mapped into another. Image schema is similar to conceptual metaphor and is also related to metaphoric gesture. Our main goal is to generate metaphoric …
Representation LearningOn Commonsense Cues in BERT for Solving Commonsense Tasks
BERT has been used for solving commonsense tasks such as CommonsenseQA. While prior research has found that BERT does contain commonsense information to some extent, there has been work showing that pre-trained models ca…
Sentiment AnalysisSentiment ClassificationBeyond Language: Learning Commonsense from Images for Reasoning
This paper proposes a novel approach to learn commonsense from images, instead of limited raw texts or costly constructed knowledge bases, for the commonsense reasoning problem in NLP. Our motivation comes from the fact …
Language ModelingLanguage ModellingLayout GenerationQuestion AnsweringContext-Gloss Augmentation for Improving Arabic Target Sense Verification
Arabic language lacks semantic datasets and sense inventories. The most common semantically-labeled dataset for Arabic is the ArabGlossBERT, a relatively small dataset that consists of 167K context-gloss pairs (about 60K…
POS