PromptBERT: Improving BERT Sentence Embeddings with Prompts
The poor performance of the original BERT for sentence semantic similarity has been widely discussed in previous works. We find that unsatisfactory performance is mainly due to the static token embeddings biases and the ineffective BERT layers, rather than the high cosine similarity of the sentence embeddings. To this end, we propose a prompt based sentence embeddings method which can reduce token embeddings biases and make the original BERT layers more effectively. By reformulating the sentence embeddings task as the fillin-the-blanks problem, our method significantly improves the performance of original BERT. We discuss two prompt representing methods and three prompt searching methods for prompt based sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised setting. For experiments, we evaluate our method on both non fine-tuned and fine-tuned settings. Even a non fine-tuned method can outperform the fine-tuned methods like unsupervised ConSERT on STS tasks. Our fine-tuned method outperforms the state-of-the-art method SimCSE in both unsupervised and supervised settings. Compared to SimCSE, we achieve 2.29 and 2.58 points improvements on BERT and RoBERTa respectively under the unsupervised setting.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingSemantic SimilaritySemantic Textual SimilaritySentenceSentence EmbeddingsSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PromptBERT: Improving BERT Sentence Embeddings with Prompts
We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to t…
Contrastive LearningDenoisingSemantic SimilaritySemantic Textual Similarity+5AgentRec: Agent Recommendation Using Sentence Embeddings Aligned to Human Feedback
Multi-agent systems must decide which agent is the most appropriate for a given task. We propose a novel architecture for recommending which LLM agent out of many should perform a task given a natural language prompt by …
reinforcement-learningReinforcement LearningSentenceSentence EmbeddingsDeep Continuous Prompt for Contrastive Learning of Sentence Embeddings
The performance of sentence representation has been remarkably improved by the framework of contrastive learning. However, recent works still require full fine-tuning, which is quite inefficient for large-scaled pre-trai…
Contrastive LearningLanguage ModelingLanguage ModellingSentence+1Dual-View Distilled BERT for Sentence Embedding
Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings,…
SentenceSentence EmbeddingSentence-EmbeddingSentence Embeddings+1Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are …
ClusteringLinear-Probe ClassificationSemantic SimilaritySemantic Textual Similarity+6