Regularly varying representation for sentence embedding
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeSentenceSentence EmbeddingSentence-EmbeddingText GenerationSimilar Papers 제목 키워드 기반
Grammatical information in BERT sentence embeddings as two-dimensional arrays
Sentence embeddings induced with various transformer architectures encode much semantic and syntactic information in a distributed manner in a one-dimensional array. We investigate whether specific grammatical informatio…
Few-Shot LearningSentenceSentence EmbeddingsUnsupervised Sentence-embeddings by Manifold Approximation and Projection
The concept of unsupervised universal sentence encoders has gained traction recently, wherein pre-trained models generate effective task-agnostic fixed-dimensional representations for phrases, sentences and paragraphs. S…
SentenceSentence Embeddingstext-classificationText Classification+1Comparative Study of Sentence Embeddings for Contextual Paraphrasing
Paraphrasing is an important aspect of natural-language generation that can produce more variety in the way specific content is presented. Traditionally, paraphrasing has been focused on finding different words that conv…
ClusteringSentenceSentence EmbeddingsText GenerationRepresentation of Ambiguity in Pre-Trained Sentence Embeddings
Pre-trained language models have been shown to be very effective for various NLP tasks. All of these models are trained on different datasets and often have different architectures. Simultaneously, various approaches for…
SentenceSentence EmbeddingsCDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word d…
Representation Learning