Coming to its senses: Lessons learned from Approximating Retrofitted BERT representations for Word Sense information
Retrofitting static vector space word representations using external knowledge bases has yielded substantial improvements in their lexical-semantic capacities but is non-trivial to apply to contextual word embeddings (CWE). In this paper, we propose MAKESENSE, a method that 'approximates' retrofitting in CWEs to better infer word sense knowledge from word contexts. We specifically analyze BERT and MAKESENSE-transformed BERT representations over a diverse set of experiments encompassing sense-sensitive similarities, alignment with human-elicited similarity judgments, and probing tasks focusing on sense distinctions and hypernymy. Our findings indicate that MAKESENSE imparts substantial improvements in word sense information over vanilla CWEs but largely preserves more complex usage of sense and directionally sensitive information such as hypernymy.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Emotion Enriched Retrofitted Word Embeddings
Word embeddings learned using the distributional hypothesis (e.g., GloVe, Word2vec) are good at encoding various lexical-semantic relations. However, they do not capture the emotion aspects of words. We present a novel r…
Sarcasm DetectionSentiment AnalysisWord EmbeddingsBERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology
An important question concerning contextualized word embedding (CWE) models like BERT is how well they can represent different word senses, especially those in the long tail of uncommon senses. Rather than build a WSD sy…
RetrievalData Science Education in Undergraduate Physics: Lessons Learned from a Community of Practice
It is becoming increasingly important that physics educators equip their students with the skills to work with data effectively. However, many educators may lack the necessary training and expertise in data science to te…
How (Not) to Incentivize Sustainable Mobility? Lessons from a Swiss Mobility Competition
We investigate the impact of a gamified experiment designed to promote sustainable mobility among students and staff members of a Swiss higher-education institution. Despite transportation being a major contributor to do…
The Human Cell Atlas & Equity: Lessons Learned
The Human Cell Atlas has been undergoing a massive effort to support global scientific equity. The co-leaders of its Equity Working Group share some lessons learned in the process.