paper-with-me

홈 › Papers

Negative Pre-aware for Noisy Cross-modal Matching

2023-12-10 · Xu Zhang, Hao Li, Mang Ye

Cross-modal noise-robust learning is a challenging task since noisy correspondence is hard to recognize and rectify. Due to the cumulative and unavoidable negative impact of unresolved noise, existing methods cannot maintain a stable performance when the noise increases. In this paper, we present a novel Negative Pre-aware Cross-modal (NPC) matching solution for large visual-language model fine-tuning on noisy downstream tasks. It is featured in two aspects: (1) For noise recognition and resistance, previous methods usually directly filter out a noise subset, we propose to estimate the negative impact of each sample. It does not need additional correction mechanisms that may predict unreliable correction results, leading to self-reinforcing error. We assign a confidence weight to each sample according to its negative impact in the training process. This adaptively adjusts the contribution of each sample to avoid noisy accumulation. (2) For maintaining stable performance with increasing noise, we utilize the memorization effect of DNNs by maintaining a memory bank. Specifically, we apply GMM to select high-confident clean samples as the memory entry, where the memory entry is used to estimate the negative impact of each sample. Since clean samples are easier distinguished by GMM with increasing noise, the memory bank can still maintain high quality at a high noise ratio. Compared to the correction mechanism focusing on noise samples, memory bank-based estimation is more robust, which makes the model performance stable on noisy datasets. Extensive experiments demonstrate that our method significantly improves matching accuracy and performance stability at increasing noise ratio. Our approach also surpasses the state-of-the-art methods by a large margin. The code is available at: https://github.com/ZhangXu0963/NPC.

📄 PDF Abstract BibTeX arXiv:2312.05777

Code (2)

zhangxu0963/npc 공식 구현 pytorch
leolee99/osa pytorch

Tasks

Cross-modal retrieval with noisy correspondenceImage-text matchingImage-to-Text RetrievalLanguage ModellingMemorization

Similar Papers 제목 키워드 기반

PointCloud-Text Matching: Benchmark Datasets and a Baseline

2024-03-28 · Yanglin Feng, Yang Qin, Dezhong Peng, Hongyuan Zhu 외

In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching~(PTM), which aims to find the exact cross-modal instance that matches a given point-cloud query or text query. PTM could b…

Contrastive LearningRetrievalText Matching

Learning From Noisy Correspondence With Tri-Partition for Cross-Modal Matching

2023-09-22 · IEEE Transactions on Multimedia 2023 9 · Zerun Feng, Zhimin Zeng, Caili Guo, Zheng Li 외

Due to high labeling cost, it is inevitable to introduce a certain proportion of noisy correspondence into visual-text datasets, resulting in poor model robustness for cross-modal matching. Although recent methods divide…

Cross-modal retrieval with noisy correspondenceMemorizationSemantic correspondenceText Matching+1

Negative-Aware Attention Framework for Image-Text Matching

2022-01-01 · CVPR 2022 1 · Kun Zhang, Zhendong Mao, Quan Wang, Yongdong Zhang

Image-text matching, as a fundamental task, bridges the gap between vision and language. The key of this task is to accurately measure similarity between these two modalities. Prior work measuring this similarity mai…

Image-text matchingText Matchingtext similarity

Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation

2023-12-27 · Zhuohang Dang, Minnan Luo, Chengyou Jia, Guang Dai 외

Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. Recently, to alleviate expensive data collection, co-occurring pairs from the Internet are automatically harvested for tra…

Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceMemorizationModel Optimization+1

Learning with Noisy Correspondence for Cross-modal Matching

2021-12-01 · NeurIPS 2021 12 · Zhenyu Huang, guocheng niu, Xiao Liu, Wenbiao Ding 외

Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge…

Cross-Modal RetrievalCross-modal retrieval with noisy correspondenceImage-text matchingMemorization+2