Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing

2023-10-18 · Dmitry Nikolaev, Sebastian Padó

The question of what kinds of linguistic information are encoded in different layers of Transformer-based language models is of considerable interest for the NLP community. Existing work, however, has overwhelmingly focused on word-level representations and encoder-only language models with the masked-token training objective. In this paper, we present experiments with semantic structural probing, a method for studying sentence-level representations via finding a subspace of the embedding space that provides suitable task-specific pairwise distances between data-points. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks, semantic textual similarity and natural-language inference. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.

📄 PDF Abstract BibTeX arXiv:2310.11923

Code (1)

macleginn/semantic-subspaces-code 공식 구현 pytorch

Tasks

Decoder Natural Language Inference Semantic Textual SimilaritySentenceSentence Embeddings

Static Word Embeddings for Sentence Semantic Representation

2025-06-05 · Takashi Wada, Yuki Hirakawa, Ryotaro Shimizu, Takahiro Kawashima 외

We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component ana…

Contrastive Learning Knowledge Distillation Sentence Word Embeddings

Efficient Sentence Embedding via Semantic Subspace Analysis

2020-02-22 · Bin Wang, Fenxiao Chen, Yuncheng Wang, C. -C. Jay Kuo

A novel sentence embedding method built upon semantic subspace analysis, called semantic subspace sentence embedding (S3E), is proposed in this work. Given the fact that word embeddings can capture semantic relationship …

Sentence Sentence Embedding Sentence-Embedding Word Embeddings

Revisiting Language Encoding in Learning Multilingual Representations

2021-02-16 · Shengjie Luo, Kaiyuan Gao, Shuxin Zheng, Guolin Ke 외

Transformer has demonstrated its great power to learn contextual word representations for multiple languages in a single model. To process multilingual sentences in the model, a learnable vector is usually assigned to ea…

Sentence Word Embeddings

Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence

2021-10-14 · Findings (EMNLP) 2021 11 · Kelvin Lo, Yuan Jin, Weicong Tan, Ming Liu 외

This paper proposes a transformer over transformer framework, called Transformer$^2$, to perform neural text segmentation. It consists of two components: bottom-level sentence encoders using pre-trained transformers, and…

Segmentation Sentence Sentence Embeddings Text Segmentation

A Bilingual Generative Transformer for Semantic Sentence Embedding

2019-11-10 · EMNLP 2020 11 · John Wieting, Graham Neubig, Taylor Berg-Kirkpatrick

Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal …