Efficient Sentence Embedding via Semantic Subspace Analysis
A novel sentence embedding method built upon semantic subspace analysis, called semantic subspace sentence embedding (S3E), is proposed in this work. Given the fact that word embeddings can capture semantic relationship while semantically similar words tend to form semantic groups in a high-dimensional embedding space, we develop a sentence representation scheme by analyzing semantic subspaces of its constituent words. Specifically, we construct a sentence model from two aspects. First, we represent words that lie in the same semantic group using the intra-group descriptor. Second, we characterize the interaction between multiple semantic groups with the inter-group descriptor. The proposed S3E method is evaluated on both textual similarity tasks and supervised tasks. Experimental results show that it offers comparable or better performance than the state-of-the-art. The complexity of our S3E method is also much lower than other parameterized models.
Code (1)
Tasks
SentenceSentence EmbeddingSentence-EmbeddingWord EmbeddingsSimilar Papers 제목 키워드 기반
Parameter-free Sentence Embedding via Orthogonal Basis
We propose a simple and robust non-parameterized approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word an…
SentenceSentence EmbeddingSentence-EmbeddingWord EmbeddingsZero-training Sentence Embedding via Orthogonal Basis
We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and it…
SentenceSentence EmbeddingSentence-EmbeddingWord EmbeddingsLocality Preserving Sentence Encoding
Although researches on word embeddings have made great progress in recent years, many tasks in natural language processing are on the sentence level. Thus, it is essential to learn sentence embeddings. Recently, Sentence…
Semantic SimilaritySemantic Textual SimilaritySentenceSentence Classification+5Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing
The question of what kinds of linguistic information are encoded in different layers of Transformer-based language models is of considerable interest for the NLP community. Existing work, however, has overwhelmingly focu…
DecoderNatural Language InferenceSemantic Textual SimilaritySentence+1Subspace Representations for Soft Set Operations and Sentence Similarities
In the field of natural language processing (NLP), continuous vector representations are crucial for capturing the semantic meanings of individual words. Yet, when it comes to the representations of sets of words, the co…
RetrievalSemantic Textual SimilaritySentenceSentence Similarity