Explaining and Improving BERT Performance on Lexical Semantic Change Detection

2021-03-12 · EACL 2021 2 · Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, Sabine Schulte im Walde

Type- and token-based embedding architectures are still competing in lexical semantic change detection. The recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models on a variety of other NLP tasks does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word, which is encoded even in the higher layers of BERT representations. By reducing the influence of orthography we considerably improve BERT's performance.

📄 PDF Abstract BibTeX arXiv:2103.07259

Code (0)

등록된 구현이 없습니다.

Tasks

Change Detection

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Attention 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Weight Decay 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not outperform SGNS on Semantic Change Detection

2020-11-14 · Severin Laicher, Gioia Baldissin, Enrique Castañeda, Dominik Schlechtweg 외

We present the results of our participation in the DIACR-Ita shared task on lexical semantic change detection for Italian. We exploit Average Pairwise Distance of token-based BERT embeddings between time points and rank …

Change Detection

DiaSense at SemEval-2020 Task 1: Modeling Sense Change via Pre-trained BERT Embeddings

2020-12-01 · SEMEVAL 2020 · Christin Beck

This paper describes DiaSense, a system developed for Task 1 {`}Unsupervised Lexical Semantic Change Detection{'} of SemEval 2020. In DiaSense, contextualized word embeddings are used to model word sense changes. This al…

Change Detection Word Embeddings

black[LSCDiscovery shared task] UAlberta at LSCDiscovery: Lexical Semantic Change Detection via Word Sense Disambiguation

2022-05-01 · LChange (ACL) 2022 5 · Daniela Teodorescu, Spencer von der Ohe, Grzegorz Kondrak

We describe our two systems for the shared task on Lexical Semantic Change Discovery in Spanish. For binary change detection, we frame the task as a word sense disambiguation (WSD) problem. We derive sense frequency dist…

Change Detection Word Sense Disambiguation

HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis

2022-02-08 · Wenjun Qiu, Yang Xu

Contextualized word embeddings have demonstrated state-of-the-art performance in various natural language processing tasks including those that concern historical semantic change. However, language models such as BERT wa…

Language Modeling Language Modelling Word Embeddings Word Similarity

black[LSCDiscovery shared task] HSE at LSCDiscovery in Spanish: Clustering and Profiling for Lexical Semantic Change Discovery

2022-05-01 · LChange (ACL) 2022 5 · Kseniia Kashleva, Alexander Shein, Elizaveta Tukhtina, Svetlana Vydrina

This paper describes the methods used for lexical semantic change discovery in Spanish. We tried the method based on BERT embeddings with clustering, the method based on grammatical profiles and the grammatical profiles …

Change Detection Clustering