Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding
Temporal language grounding (TLG) aims to localize a video segment in an untrimmed video based on a natural language description. To alleviate the expensive cost of manual annotations for temporal boundary labels, we are dedicated to the weakly supervised setting, where only video-level descriptions are provided for training. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. However, the temporal structure of the video as well as the complicated semantics in the sentence are lost during the learning. In this work, we propose a novel candidate-free framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG. Instead of view the sentence and candidate moments as a whole, FSAN learns token-by-clip cross-modal semantic alignment by an iterative cross-modal interaction module, generates a fine-grained cross-modal semantic alignment map, and performs grounding directly on top of the map. Extensive experiments are conducted on two widely-used benchmarks: ActivityNet-Captions, and DiDeMo, where our FSAN achieves state-of-the-art performance.
Code (0)
등록된 구현이 없습니다.
Tasks
cross-modal alignmentSentenceSimilar Papers 제목 키워드 기반
AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding
Weakly supervised visual grounding (VG) aims to locate objects in images based on text descriptions. Despite significant progress, existing methods lack strong cross-modal reasoning to distinguish subtle semantic differe…
Contrastive LearningVisual GroundingLearning to Tell Apart: Weakly Supervised Video Anomaly Detection via Disentangled Semantic Alignment
Recent advancements in weakly-supervised video anomaly detection have achieved remarkable performance by applying the multiple instance learning paradigm based on multimodal foundation models such as CLIP to highlight an…
Weakly-supervised Video Anomaly DetectionMultiple Instance LearningContrastive LearningExploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels aims to achieve pixel-level predictions using Class Activation Maps (CAMs). Recently, Contrastive Language-Image Pre-training (CLIP) has been introdu…
AttributeSemantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationVadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection
The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics.…
Anomaly DetectionBinary ClassificationVideo Anomaly DetectionWeakly-supervised Video Anomaly DetectionWeakly-supervised Object Localization for Few-shot Learning and Fine-grained Few-shot Learning
Few-shot learning (FSL) aims to learn novel visual categories from very few samples, which is a challenging problem in real-world applications. Many methods of few-shot classification work well on general images to learn…
ClassificationFew-Shot LearningGeneral ClassificationObject Localization+1