RepAL: A Simple and Plug-and-play Method for Improving Unsupervised Sentence Representations
Unsupervised sentence representation learning is a fundamental problem in natural language processing and has been studied extensively in recent years. This paper presents Representation ALchemy (RepAL), an extremely simple post-processing method that enhances unsupervised sentence representations. The basic idea in RepAL is to extract redundant information from the representation of a sentence generated by the existing models and then refine the representation through an embedding refinement operation to filter such redundant information. In this paper, we analyze the redundant information from two levels: sentence-level and corpus-level, and the theoretical analysis for the latter is also conducted. We point out that RepAL is free of training and is a plug-and-play method that can be combined with most existing unsupervised sentence learning models. Extensive experiments demonstrate RepAL’s effectiveness and show that RepAL is a model-agnostic method for unsupervised sentence embedding enhancement. Besides, we also designed detailed ablation studies to understand why RepAL works and provided in-depth analysis and understanding of the redundant information.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSentenceSentence EmbeddingSentence-EmbeddingSimilar Papers 제목 키워드 기반
A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancement
Generating proper embedding of sentences through an unsupervised way is beneficial to semantic matching and retrieval problems in real-world scenarios. This paper presents Representation ALchemy (RepAL), an extremely sim…
RetrievalSentenceSentence EmbeddingSentence-EmbeddingAlleviating Over-smoothing for Unsupervised Sentence Representation
Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning hav…
Contrastive LearningSemantic Textual SimilaritySentenceA Plug-and-Play Method for Controlled Text Generation
Large pre-trained language models have repeatedly shown their ability to produce fluent text. Yet even when starting from a prompt, generation can continue in many plausible directions. Current decoding methods with the …
SentenceStory GenerationText GenerationMake Compound Sentences Simple to Analyze: Learning to Split Sentences for Aspect-based Sentiment Analysis
In the domain of Aspect-Based Sentiment Analysis (ABSA), generative methods have shown promising results and achieved substantial advancements. However, despite these advancements, the tasks of extracting sentiment quadr…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)SentenceSentiment AnalysisUnsupervised Sentence Representation Learning with Frequency-induced Adversarial Tuning and Incomplete Sentence Filtering
Pre-trained Language Model (PLM) is nowadays the mainstay of Unsupervised Sentence Representation Learning (USRL). However, PLMs are sensitive to the frequency information of words from their pre-training corpora, result…
Language ModellingRepresentation LearningSentenceSentence Embeddings