NAC: Mitigating Noisy Correspondence in Cross-Modal Matching Via Neighbor Auxiliary Corrector
The presence of noisy correspondence within cross-modal matching has significantly undermined the performance of existing matching methods. In this paper, we introduce a robust framework named Neighbor Auxiliary Corrector (NAC) for alleviating noise by utilizing the neighbors, which are indicative of similar textual targets. NAC is inspired by an observation that similar texts tend to correspond to similar images. Leveraging the zero-shot capabilities of Pre-trained Language Models (PLMs), we identify the top-k nearest neighbors for each positive image-text pair. Subsequently, the side information provided by these neighbors is harnessed for both sample verification and sample rectification. Extensive experiments on benchmark datasets demonstrate that our framework can significantly boost the performance and is more robust to various levels of noisy correspondence.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-modal retrieval with noisy correspondenceSimilar Papers 제목 키워드 기반
Learning with Noisy Correspondence for Cross-modal Matching
Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge…
Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceImage-text matchingMemorization+2Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
Noisy correspondence that refers to mismatches in cross-modal data pairs, is prevalent on human-annotated or web-crawled datasets. Prior approaches to leverage such data mainly consider the application of uni-modal noisy…
Cross-modal retrieval with noisy correspondenceUnsupervised Visible-Infrared Person Re-Identification via Progressive Graph Matching and Alternate Learning
Unsupervised visible-infrared person re-identification is a challenging task due to the large modality gap and the unavailability of cross-modality correspondences. Cross-modality correspondences are very crucial to …
Contrastive LearningGraph MatchingPerson Re-IdentificationREPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy Correspondence
The presence of noise in acquired data invariably leads to performance degradation in cross-modal matching. Unfortunately, obtaining precise annotations in the multimodal field is expensive, which has prompted some metho…
Cross-modal retrieval with noisy correspondenceUGNCL: Uncertainty-Guided Noisy Correspondence Learning for Efficient Cross-Modal Matching
Cross-modal matching has recently gained significant popularity to facilitate retrieval across multi-modal data, and existing works are highly relied on an implicit assumption that the training data pairs are perfectly a…
Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceImage-text matchingImage-text Retrieval