SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fusion information across layers to find better sentence representation. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentence-level probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available.
Code (3)
Tasks
Semantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation
ABSTRACT A sentence embedding vector can be obtained by connecting a global average pooling (GAP) to a pre-trained language model. The problem of such a sentence embedding vector using a GAP is that it is generated wit…
Language ModelingLanguage ModellingSemantic Textual SimilaritySentence+3Word Embeddings Revisited: Do LLMs Offer Something New?
Learning meaningful word embeddings is key to training a robust language model. The recent rise of Large Language Models (LLMs) has provided us with many new word/sentence/document embedding models. Although LLMs have sh…
Document EmbeddingLanguage ModelingLanguage ModellingSentence+1Locality Preserving Sentence Encoding
Although researches on word embeddings have made great progress in recent years, many tasks in natural language processing are on the sentence level. Thus, it is essential to learn sentence embeddings. Recently, Sentence…
Semantic SimilaritySemantic Textual SimilaritySentenceSentence Classification+5Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents
Encoder models trained for the embedding of sentences or short documents have proven useful for tasks such as semantic search and topic modeling. In this paper, we present a version of the SwissBERT encoder model that we…
ArticlesContrastive LearningRetrievaltext-classification+1Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are …
ClusteringLinear-Probe ClassificationSemantic SimilaritySemantic Textual Similarity+6