Deep Continuous Prompt for Contrastive Learning of Sentence Embeddings
The performance of sentence representation has been remarkably improved by the framework of contrastive learning. However, recent works still require full fine-tuning, which is quite inefficient for large-scaled pre-trained language models. To this end, we present a novel method which freezes the whole language model and only optimizes the prefix deep continuous prompts. It not only tunes around 0.1\% parameters of the original language model, but avoids the cumbersome computation of searching handcrafted prompts. Experimental results show that our proposed DCPCSE outperforms the state-of-the-art method SimCSE by a large margin. We raise the performance of unsupervised BERT$_{base}$ and supervised RoBERTa$_{large}$ by 2.24 and 1.00 points, respectively. Our code will be released at Github.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingSentenceSentence EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
D2CSE: Difference-aware Deep continuous prompts for Contrastive Sentence Embeddings
This paper describes Difference-aware Deep continuous prompt for Contrastive Sentence Embeddings (D2CSE) that learns sentence embeddings. Compared to state-of-the-art approaches, D2CSE computes sentence vectors that are …
Contrastive LearningRetrievalSemantic Textual SimilaritySentence+2Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning
Contrastive learning has been demonstrated to be effective in enhancing pre-trained language models (PLMs) to derive superior universal sentence embeddings. However, existing contrastive methods still have two limitation…
AttributeContrastive LearningSemantic Textual SimilaritySentence+2PromptBERT: Improving BERT Sentence Embeddings with Prompts
We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to t…
Contrastive LearningDenoisingSemantic SimilaritySemantic Textual Similarity+5Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding
Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In…
ClusteringContrastive LearningData AugmentationSemantic Textual Similarity+4Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering
Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to enc…
Prompt EngineeringSemantic Textual SimilaritySentenceSentence Embedding+3