Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization
Weakly supervised dense object localization (WSDOL) relies generally on Class Activation Mapping (CAM), which exploits the correlation between the class weights of the image classifier and the pixel-level features. Due to the limited ability to address intra-class variations, the image classifier cannot properly associate the pixel features, leading to inaccurate dense localization maps. In this paper, we propose to explicitly construct multi-modal class representations by leveraging the Contrastive Language-Image Pre-training (CLIP), to guide dense localization. More specifically, we propose a unified transformer framework to learn two-modalities of class-specific tokens, i.e., class-specific visual and textual tokens. The former captures semantics from the target visual data while the latter exploits the class-related language priors from CLIP, providing complementary information to better perceive the intra-class diversities. In addition, we propose to enrich the multi-modal class-specific tokens with sample-specific contexts comprising visual context and image-language context. This enables more adaptive class representation learning, which further facilitates dense localization. Extensive experiments show the superiority of the proposed method for WSDOL on two multi-label datasets, i.e., PASCAL VOC and MS COCO, and one single-label dataset, i.e., OpenImages. Our dense localization maps also lead to the state-of-the-art weakly supervised semantic segmentation (WSSS) results on PASCAL VOC and MS COCO.
Code (0)
등록된 구현이 없습니다.
Tasks
Object LocalizationRepresentation LearningSemantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-scale Bottleneck Transformer for Weakly Supervised Multimodal Violence Detection
Weakly supervised multimodal violence detection aims to learn a violence detection model by leveraging multiple modalities such as RGB, optical flow, and audio, while only video-level annotations are available. In the pu…
Anomaly Detection In Surveillance VideosOptical Flow EstimationMulti-class Token Transformer for Weakly Supervised Semantic Segmentation
This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions o…
ObjectObject LocalizationSemantic SegmentationWeakly supervised Semantic Segmentation+1Class Tokens Infusion for Weakly Supervised Semantic Segmentation
Weakly Supervised Semantic Segmentation (WSSS) relies on Class Activation Maps (CAMs) to extract spatial information from image-level labels. With the success of Vision Transformer (ViT) the migration of ViT is activ…
Semantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationEnhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance
Weakly supervised multimodal video anomaly detection has gained significant attention, yet the potential of the text modality remains under-explored. Text provides explicit semantic information that can enhance anomaly c…
Video Anomaly DetectionMCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation
This paper proposes a novel transformer-based framework that aims to enhance weakly supervised semantic segmentation (WSSS) by generating accurate class-specific object localization maps as pseudo labels. Building upon t…
Object LocalizationSemantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic Segmentation