PromptBERT: Improving BERT Sentence Embeddings with Prompts
We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.
Code (1)
Tasks
Contrastive LearningDenoisingSemantic SimilaritySemantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingSentence EmbeddingsSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PromptBERT: Improving BERT Sentence Embeddings with Prompts
The poor performance of the original BERT for sentence semantic similarity has been widely discussed in previous works. We find that unsatisfactory performance is mainly due to the static token embeddings biases and th…
DenoisingSemantic SimilaritySemantic Textual SimilaritySentence+2AgentRec: Agent Recommendation Using Sentence Embeddings Aligned to Human Feedback
Multi-agent systems must decide which agent is the most appropriate for a given task. We propose a novel architecture for recommending which LLM agent out of many should perform a task given a natural language prompt by …
reinforcement-learningReinforcement LearningSentenceSentence EmbeddingsDeep Continuous Prompt for Contrastive Learning of Sentence Embeddings
The performance of sentence representation has been remarkably improved by the framework of contrastive learning. However, recent works still require full fine-tuning, which is quite inefficient for large-scaled pre-trai…
Contrastive LearningLanguage ModelingLanguage ModellingSentence+1Dual-View Distilled BERT for Sentence Embedding
Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings,…
SentenceSentence EmbeddingSentence-EmbeddingSentence Embeddings+1Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are …
ClusteringLinear-Probe ClassificationSemantic SimilaritySemantic Textual Similarity+6