Generalised Spherical Text Embedding
This paper aims to provide an unsupervised modelling approach that allows for a more flexible representation of text embeddings. It jointly encodes the words and the paragraphs as individual matrices of arbitrary column dimension with unit Frobenius norm. The representation is also linguistically motivated with the introduction of a novel similarity metric. The proposed modelling and the novel similarity metric exploits the matrix structure of embeddings. We then go on to show that the same matrices can be reshaped into vectors of unit norm and transform our problem into an optimization problem over the spherical manifold. We exploit manifold optimization to efficiently train the matrix embeddings. We also quantitatively verify the quality of our text embeddings by showing that they demonstrate improved results in document classification, document clustering, and semantic textual similarity benchmark tests.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringDocument ClassificationSemantic Textual SimilaritySimilar Papers 제목 키워드 기반
Spectral clustering under degree heterogeneity: a case for the random walk Laplacian
This paper shows that graph spectral embedding using the random walk Laplacian produces vector representations which are completely corrected for node degree. Under a generalised random dot product graph, the embedding p…
ClusteringStochastic Block ModelSpherical Text Embedding
Unsupervised text embedding has shown great power in a wide range of NLP tasks. While text embeddings are typically learned in the Euclidean space, directional similarity is often more effective in tasks such as word sim…
ClusteringRiemannian optimizationWord SimilarityDifferentiable and accelerated spherical harmonic and Wigner transforms
Many areas of science and engineering encounter data defined on spherical manifolds. Modelling and analysis of spherical data often necessitates spherical harmonic transforms, at high degrees, and increasingly requires e…
SphereRE: Distinguishing Lexical Relations with Hyperspherical Relation Embeddings
Lexical relations describe how meanings of terms relate to each other. Typical examples include hypernymy, synonymy, meronymy, etc. Automatic distinction of lexical relations is vital for NLP applications, and also chall…
RelationRepresentation LearningGeometry-Preserving Aggregation for Mixture-of-Experts Embedding Models
Mixture-of-Experts (MoE) embedding models combine expert outputs using weighted linear summation, implicitly assuming a linear subspace structure in the embedding space. This assumption is shown to be inconsistent with t…
Semantic Similarity