Achieving Semantic Consistency: Contextualized Word Representations for Political Text Analysis

2024-12-03 · Ruiyu Zhang, Lin Nie, Ce Zhao, Qingyang Chen

Accurately interpreting words is vital in political science text analysis; some tasks require assuming semantic stability, while others aim to trace semantic shifts. Traditional static embeddings, like Word2Vec effectively capture long-term semantic changes but often lack stability in short-term contexts due to embedding fluctuations caused by unbalanced training data. BERT, which features transformer-based architecture and contextual embeddings, offers greater semantic consistency, making it suitable for analyses in which stability is crucial. This study compares Word2Vec and BERT using 20 years of People's Daily articles to evaluate their performance in semantic representations across different timeframes. The results indicate that BERT outperforms Word2Vec in maintaining semantic stability and still recognizes subtle semantic variations. These findings support BERT's use in text analysis tasks that require stability, where semantic changes are not assumed, offering a more reliable foundation than static alternatives.

📄 PDF Abstract BibTeX arXiv:2412.04505

Code (0)

등록된 구현이 없습니다.

Tasks

Articles

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Multi-Head Attention 설명 없음

Weight Decay 설명 없음

WordPiece 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings

2022-08-20 · Yile Wang, Yue Zhang

Contextualized word embeddings in language models have given much advance to NLP. Intuitively, sentential information is integrated into the representation of words, which can help model polysemy. However, context sensit…

Position Sentence Word Embeddings Word Sense Disambiguation

Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction

2021-05-27 · Findings (ACL) 2021 8 · Kosuke Yamada, Ryohei Sasano, Koichi Takeda

Contextualized word representations have proven useful for various natural language processing tasks. However, it remains unclear to what extent these representations can cover hand-coded semantic information such as sem…

Clustering

Unsupervised Distillation of Syntactic Information from Contextualized Word Representations

2020-10-11 · EMNLP (BlackboxNLP) 2020 11 · Shauli Ravfogel, Yanai Elazar, Jacob Goldberger, Yoav Goldberg

Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and syntactic tasks. In this work, we tackle the task of unsupervised disentanglement between semantics and struc…

Disentanglement Metric Learning

Learning Taxonomies of Concepts and not Words using Contextualized Word Representations: A Position Paper

2019-01-31 · Lukas Schmelzeisen, Steffen Staab

Taxonomies are semantic hierarchies of concepts. One limitation of current taxonomy learning systems is that they define concepts as single words. This position paper argues that contextualized word representations, whic…

Position

The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations

2024-02-22 · Aina Garí Soler, Matthieu Labeau, Chloé Clavel

When deriving contextualized word representations from language models, a decision needs to be made on how to obtain one for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represe…