Negative Pre-aware for Noisy Cross-modal Matching
Cross-modal noise-robust learning is a challenging task since noisy correspondence is hard to recognize and rectify. Due to the cumulative and unavoidable negative impact of unresolved noise, existing methods cannot maintain a stable performance when the noise increases. In this paper, we present a novel Negative Pre-aware Cross-modal (NPC) matching solution for large visual-language model fine-tuning on noisy downstream tasks. It is featured in two aspects: (1) For noise recognition and resistance, previous methods usually directly filter out a noise subset, we propose to estimate the negative impact of each sample. It does not need additional correction mechanisms that may predict unreliable correction results, leading to self-reinforcing error. We assign a confidence weight to each sample according to its negative impact in the training process. This adaptively adjusts the contribution of each sample to avoid noisy accumulation. (2) For maintaining stable performance with increasing noise, we utilize the memorization effect of DNNs by maintaining a memory bank. Specifically, we apply GMM to select high-confident clean samples as the memory entry, where the memory entry is used to estimate the negative impact of each sample. Since clean samples are easier distinguished by GMM with increasing noise, the memory bank can still maintain high quality at a high noise ratio. Compared to the correction mechanism focusing on noise samples, memory bank-based estimation is more robust, which makes the model performance stable on noisy datasets. Extensive experiments demonstrate that our method significantly improves matching accuracy and performance stability at increasing noise ratio. Our approach also surpasses the state-of-the-art methods by a large margin. The code is available at: https://github.com/ZhangXu0963/NPC.
Code (2)
Tasks
Cross-modal retrieval with noisy correspondenceImage-text matchingImage-to-Text RetrievalLanguage ModellingMemorizationSimilar Papers 제목 키워드 기반
PointCloud-Text Matching: Benchmark Datasets and a Baseline
In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching~(PTM), which aims to find the exact cross-modal instance that matches a given point-cloud query or text query. PTM could b…
Contrastive LearningRetrievalText MatchingLearning From Noisy Correspondence With Tri-Partition for Cross-Modal Matching
Due to high labeling cost, it is inevitable to introduce a certain proportion of noisy correspondence into visual-text datasets, resulting in poor model robustness for cross-modal matching. Although recent methods divide…
Cross-modal retrieval with noisy correspondenceMemorizationSemantic correspondenceText Matching+1Negative-Aware Attention Framework for Image-Text Matching
Image-text matching, as a fundamental task, bridges the gap between vision and language. The key of this task is to accurately measure similarity between these two modalities. Prior work measuring this similarity mai…
Image-text matchingText Matchingtext similarityNoisy Correspondence Learning with Self-Reinforcing Errors Mitigation
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. Recently, to alleviate expensive data collection, co-occurring pairs from the Internet are automatically harvested for tra…
Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceMemorizationModel Optimization+1Learning with Noisy Correspondence for Cross-modal Matching
Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge…
Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceImage-text matchingMemorization+2