Knowledge Aided Consistency for Weakly Supervised Phrase Grounding
Given a natural language query, a phrase grounding system aims to localize mentioned objects in an image. In weakly supervised scenario, mapping between image regions (i.e., proposals) and language is not available in the training set. Previous methods address this deficiency by training a grounding system via learning to reconstruct language information contained in input queries from predicted proposals. However, the optimization is solely guided by the reconstruction loss from the language modality, and ignores rich visual information contained in proposals and useful cues from external knowledge. In this paper, we explore the consistency contained in both visual and language modalities, and leverage complementary external knowledge to facilitate weakly supervised grounding. We propose a novel Knowledge Aided Consistency Network (KAC Net) which is optimized by reconstructing input query and proposal's information. To leverage complementary knowledge contained in the visual features, we introduce a Knowledge Based Pooling (KBP) gate to focus on query-related proposals. Experiments show that KAC Net provides a significant improvement on two popular datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Phrase GroundingSimilar Papers 제목 키워드 기반
PIRC Net : Using Proposal Indexing, Relationships and Context for Phrase Grounding
Phrase Grounding aims to detect and localize objects in images that are referred to and are queried by natural language phrases. Phrase grounding finds applications in tasks such as Visual Dialog, Visual Search and Image…
Phrase GroundingSentenceTransfer LearningVisual DialogImproving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
Weakly supervised phrase grounding aims at learning region-phrase correspondences using only image-sentence pairs. A major challenge thus lies in the missing links between image regions and sentence phrases during traini…
Contrastive LearningKnowledge DistillationObjectobject-detection+4MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Phrase localization is a task that studies the mapping from textual phrases to regions of an image. Given difficulties in annotating phrase-to-object datasets at scale, we develop a Multimodal Alignment Framework (MAF) t…
Phrase GroundingLearning to Selectively Learn for Weakly-supervised Paraphrase Generation
Paraphrase generation is a longstanding NLP task that has diverse applications for downstream NLP tasks. However, the effectiveness of existing efforts predominantly relies on large amounts of golden labeled data. Though…
Language ModelingLanguage ModellingMeta-LearningParaphrase Generation+1Weakly Supervised 3D Segmentation via Receptive-driven Pseudo Label Consistency and Structural Consistency
As manual point-wise label is time and labor-intensive for fully supervised large-scale point cloud semantic segmentation, weakly supervised method is increasingly active. However, existing methods fail to generate high-…
Pseudo LabelSegmentationSemantic Segmentation