Exploiting Pseudo Image Captions for Multimodal Summarization
Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives. In this paper, we study this problem from the perspective of Mutual Information (MI) optimization. It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist. Guided by a more general lower bound form for optimization, we propose a contrastive learning strategy regulated by progressively refined cross-modal similarity, to more accurately optimize MI between an image/text anchor and its negative texts/images instead of improperly minimizing it. Our method performs competitively on four downstream cross-modal tasks and systematically balances the beneficial and harmful effects of (partial) false negative samples under theoretical guidance.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningContrastive LearningImage CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MMCIG: Multimodal Cover Image Generation for Text-only Documents and Its Dataset Construction via Pseudo-labeling
In this study, we introduce a novel cover image generation task that produces both a concise summary and a visually corresponding image from a given text-only document. Because no existing datasets are available for this…
Image GenerationUniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
With the rapid increase of multimedia data, a large body of literature has emerged to work on multimodal summarization, the majority of which target at refining salient information from textual and visual modalities to o…
Abstractive Text SummarizationDecoderImage CaptioningKnowledge Distillation+1SD-RSIC: Summarization Driven Deep Remote Sensing Image Captioning
Deep neural networks (DNNs) have been recently found popular for image captioning problems in remote sensing (RS). Existing DNN based approaches rely on the availability of a training set made up of a high number of RS i…
Image CaptioningBioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a …
Image RetrievalMM-AVS: A Full-Scale Dataset for Multi-modal Summarization
Multimodal summarization becomes increasingly significant as it is the basis for question answering, Web search, and many other downstream tasks. However, its learning materials have been lacking a holistic organization …
Question Answering