Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

2021-10-05 · PACLIC 2021 11 · Yi Zhou, Danushka Bollegala

Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static word embeddings such as GloVe represent words by relatively low-dimensional, memory- and compute-efficient vectors but are not sensitive to the different senses of the word. We propose Context Derived Embeddings of Senses (CDES), a method that extracts sense related information from contextualised embeddings and injects it into static embeddings to create sense-specific static embeddings. Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sense-specific static embeddings reporting comparable performance to the current state-of-the-art sense embeddings.

📄 PDF Abstract BibTeX arXiv:2110.02204

Code (0)

등록된 구현이 없습니다.

Tasks

Word EmbeddingsWord Sense Disambiguation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

WordPiece 설명 없음

Adam 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Residual Connection 설명 없음

Weight Decay 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

2022-03-14 · Yi Zhou, Masahiro Kaneko, Danushka Bollegala

Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to …

Word Embeddings

Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

Code (0)

Tasks

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반

Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings

Sense Embeddings are also Biased -- Evaluating Social Biases in Static and Contextualised Sense Embeddings

Assessing Polyseme Sense Similarity through Co-predication Acceptability and Contextualised Embedding Distance