ELMo and BERT in semantic change detection for Russian

2020-10-07 · Julia Rodina, Yuliya Trofimova, Andrey Kutuzov, Ekaterina Artemova

We study the effectiveness of contextualized embeddings for the task of diachronic semantic change detection for Russian language data. Evaluation test sets consist of Russian nouns and adjectives annotated based on their occurrences in texts created in pre-Soviet, Soviet and post-Soviet time periods. ELMo and BERT architectures are compared on the task of ranking Russian words according to the degree of their semantic change over time. We use several methods for aggregation of contextualized embeddings from these architectures and evaluate their performance. Finally, we compare unsupervised and supervised techniques in this task.

📄 PDF Abstract BibTeX arXiv:2010.03481

Code (0)

등록된 구현이 없습니다.

Tasks

Change Detection

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Tanh Activation 설명 없음

Sigmoid Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

BiLSTM A Bidirectional LSTM, or biLSTM, is a sequence processing model that consists of two LSTMs: one taking the input in a forward direction, and the other in a backwards…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Representing ELMo embeddings as two-dimensional text online

2021-03-30 · EACL 2021 2 · Andrey Kutuzov, Elizaveta Kuzmenko

We describe a new addition to the WebVectors toolkit which is used to serve word embedding models over the Web. The new ELMoViz module adds support for contextualized embedding architectures, in particular for ELMo model…

Sentence Vocal Bursts Valence Prediction

Sentence Embeddings for Russian NLU

2019-10-29 · Dmitry Popov, Alexander Pugachev, Polina Svyatokum, Elizaveta Svitanko 외

We investigate the performance of sentence embeddings models on several tasks for the Russian language. In our comparison, we include such tasks as multiple choice question answering, next sentence prediction, and paraph…

Multiple-choice Paraphrase Identification Prediction Question Answering+2

UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection

2020-04-30 · SEMEVAL 2020 · Andrey Kutuzov, Mario Giulianelli

We apply contextualised word embeddings to lexical semantic change detection in the SemEval-2020 Shared Task 1. This paper focuses on Subtask 2, ranking words by the degree of their semantic drift over time. We analyse t…

Change Detection Word Embeddings

Sentence Level Representation And Language Models In The Task Of Coreference Resolution For Russian

2019-05-29 · Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2019” 2019 5 · Le T. A., Petrov M. A., Kuratov Y. M., Burtsev M. S.

Coreference Resolution (CR) is one of the most difficult tasks in the field of Natural Language Processing due to the lack of deeply and comprehensively understanding the semantic meaning of the mention in not only the s…

coreference-resolution Coreference Resolution Sentence

Semantic Role Labeling with Pretrained Language Models for Known and Unknown Predicates

2019-09-01 · RANLP 2019 9 · Daniil Larionov, Artem Shelmanov, Elena Chistova, Ivan Smirnov

We build the first full pipeline for semantic role labelling of Russian texts. The pipeline implements predicate identification, argument extraction, argument classification (labeling), and global scoring via integer lin…

Classification General Classification Semantic Role Labeling