Embedding And Clustering Your Data Can Improve Contrastive Pretraining

2024-07-26 · Luke Merrick

Recent studies of large-scale contrastive pretraining in the text embedding domain show that using single-source minibatches, rather than mixed-source minibatches, can substantially improve overall model accuracy. In this work, we explore extending training data stratification beyond source granularity by leveraging a pretrained text embedding model and the classic k-means clustering algorithm to further split training data apart by the semantic clusters within each source. Experimentally, we observe a notable increase in NDCG@10 when pretraining a BERT-based text embedding model on query-passage pairs from the MSMARCO passage retrieval dataset. Additionally, we conceptually connect our clustering approach to both the Topic Aware Sampling (TAS) aspect of the TAS-B methodology and the nearest-neighbor-based hard-negative mining aspect of the ANCE methodology and discuss how this unified view motivates future lines of research on the organization of contrastive pretraining data.

📄 PDF Abstract BibTeX arXiv:2407.18887

Code (0)

등록된 구현이 없습니다.

Tasks

ClusteringPassage Retrieval Retrieval

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

k-Means Clustering k-Means Clustering is a clustering algorithm that divides a training set into $k$ different clusters of examples that are near each other. It works by initializing $k$…

Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

2024-03-07 · Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov 외

Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional embeddings allowing one to adjust semant…

Clustering intent-classification Intent Classification Negation+1

From Representation to Clusters: A Contrastive Learning Approach for Attributed Hypergraph Clustering

2026-03-10 · Li Ni, Shuaikang Zeng, Lin Mu, Longlong Lin arxiv

Contrastive learning has demonstrated strong performance in attributed hypergraph clustering. Typically, existing methods based on contrastive learning first learn node embeddings and then apply clustering algorithms, su…

Representation Learning Contrastive Learning

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

2024-05-30 · Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo wang 외

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information re…

Information Retrieval Retrieval Text Retrieval

Cluster Analysis with Deep Embeddings and Contrastive Learning

2021-09-26 · Ramakrishnan Sundareswaran, Jansel Herrera-Gerena, John Just, Ali Jannesari

Unsupervised disentangled representation learning is a long-standing problem in computer vision. This work proposes a novel framework for performing image clustering from deep embeddings by combining instance-level contr…

Clustering Contrastive Learning Image Clustering Representation Learning+2

Contrastive Multi-view Hyperbolic Hierarchical Clustering

2022-05-05 · Fangfei Lin, Bing Bai, Kun Bai, Yazhou Ren 외

Hierarchical clustering recursively partitions data at an increasingly finer granularity. In real-world applications, multi-view data have become increasingly important. This raises a less investigated problem, i.e., mul…

Clustering