Static Word Embeddings for Sentence Semantic Representation

2025-06-05 · Takashi Wada, Yuki Hirakawa, Ryotaro Shimizu, Takahiro Kawashima, Yuki Saito

We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even rivals a basic Sentence Transformer model (SimCSE) on some data sets. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are irrelevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.

📄 PDF Abstract BibTeX arXiv:2506.04624

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive Learning Knowledge DistillationSentenceWord Embeddings

Methods 이 논문이 사용한 방법론

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

How Can BERT Help Lexical Semantics Tasks?

2019-11-07 · Yile Wang, Leyang Cui, Yue Zhang

Contextualized embeddings such as BERT can serve as strong input representations to NLP tasks, outperforming their static embeddings counterparts such as skip-gram, CBOW and GloVe. However, such embeddings are dynamic, c…

Sentence Word Embeddings

Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm

2023-04-06 · Matteo Muffo, Roberto Tedesco, Licia Sbattella, Vincenzo Scotti

The introduction of embedding techniques has pushed forward significantly the Natural Language Processing field. Many of the proposed solutions have been presented for word-level encoding; anyhow, in the last years, new …

Semantic Textual Similarity Sentence Sentence Embedding Sentence-Embedding+1

Sentence Curve Language Models

2026-02-02 · DongNyeong Heo, Taehwan Kim, Heeyoul Choi arxiv

Language models (LMs) are a central component of modern AI systems, and diffusion language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word embeddings not only to represent th…

Knowledge Distillation

Joint Unsupervised Learning of Semantic Representation of Words and Roles in Dependency Trees

2017-09-01 · RANLP 2017 9 · Michal Konkol

In this paper, we introduce WoRel, a model that jointly learns word embeddings and a semantic representation of word relations. The model learns from plain text sentences and their dependency parse trees. The word embedd…

Named Entity Recognition (NER)Question Answering Sentence Sentiment Analysis+2

Correcting the Common Discourse Bias in Linear Representation of Sentences using Conceptors

2018-11-17 · Liu Tianlin, Sedoc João, Ungar Lyle

Distributed representations of words, better known as word embeddings, have become important building blocks for natural language processing tasks. Numerous studies are devoted to transferring the success of unsupervised…

Semantic Textual Similarity Sentence Sentence Embedding Sentence-Embedding+2