Clustering-Aware Negative Sampling for Unsupervised Sentence Representation
Contrastive learning has been widely studied in sentence representation learning. However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples. This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives. To address these issues, we propose ClusterNS (Clustering-aware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework. Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning. Our code has been made publicly available.
Code (1)
Tasks
ClusteringContrastive LearningRepresentation LearningSemantic Textual SimilaritySentenceSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
In this paper, we focus on the problem of unsupervised image-sentence matching. Existing research explores to utilize document-level structural information to sample positive and negative instances for model training. Al…
Representation LearningSemantic SimilaritySemantic Textual SimilaritySentenceESANS: Effective and Semantic-Aware Negative Sampling for Large-Scale Retrieval Systems
Industrial recommendation systems typically involve a two-stage process: retrieval and ranking, which aims to match users with millions of items. In the retrieval stage, classic embedding-based retrieval (EBR) methods de…
ClusteringDiversityRecommendation SystemsRetrievalContrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding
Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In…
ClusteringContrastive LearningData AugmentationSemantic Textual Similarity+4Fuzzy Cluster-Aware Contrastive Clustering for Time Series
The rapid growth of unlabeled time series data, driven by the Internet of Things (IoT), poses significant challenges in uncovering underlying patterns. Traditional unsupervised clustering methods often fail to capture th…
ClusteringContrastive LearningData AugmentationRepresentation Learning+1Debiased Contrastive Learning of Unsupervised Sentence Representations
Recently, contrastive learning has shown effectiveness in fine-tuning pre-trained language models (PLM) to derive sentence representations, which pulls augmented positive examples together to improve the alignment while …
Contrastive LearningSemantic Textual SimilaritySentence