Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models
A variety of contextualised language models have been proposed in the NLP community, which are trained on diverse corpora to produce numerous Neural Language Models (NLMs). However, different NLMs have reported different levels of performances in downstream NLP applications when used as text representations. We propose a sentence-level meta-embedding learning method that takes independently trained contextualised word embedding models and learns a sentence embedding that preserves the complementary strengths of the input source NLMs. Our proposed method is unsupervised and is not tied to a particular downstream task, which makes the learnt meta-embeddings in principle applicable to different tasks that require sentence representations. Specifically, we first project the token-level embeddings obtained by the individual NLMs and learn attention weights that indicate the contributions of source embeddings towards their token-level meta-embeddings. Next, we apply mean and max pooling to produce sentence-level meta-embeddings from token-level meta-embeddings. Experimental results on semantic textual similarity benchmarks show that our proposed unsupervised sentence-level meta-embedding method outperforms previously proposed sentence-level meta-embedding methods as well as a supervised baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity
We address the task of unsupervised Semantic Textual Similarity (STS) by ensembling diverse pre-trained sentence encoders into sentence meta-embeddings. We apply, extend and evaluate different meta-embedding methods from…
Dimensionality ReductionSemantic Textual SimilaritySentenceSTS+1LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hi…
Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding
Contrastive learning-based methods, such as unsup-SimCSE, have achieved state-of-the-art (SOTA) performances in learning unsupervised sentence embeddings. However, in previous studies, each embedding used for contrastive…
Contrastive LearningSemantic SimilaritySemantic Textual SimilaritySentence+5Meta-Task Prompting Elicits Embeddings from Large Language Models
We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need f…
Semantic Textual SimilaritySentenceSentence EmbeddingsSTSEvaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings
Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community due to their superior performance when representing texts in numerous downstream applications. However…
Dimensionality ReductionSentenceSentence Embeddings