Weakly Supervised Annotations for Multi-modal Greeting Cards Dataset
In recent years, there is a growing number of pre-trained models trained on a large corpus of data and yielding good performance on various tasks such as classifying multimodal datasets. These models have shown good performance on natural images but are not fully explored for scarce abstract concepts in images. In this work, we introduce an image/text-based dataset called Greeting Cards. Dataset (GCD) that has abstract visual concepts. In our work, we propose to aggregate features from pretrained images and text embeddings to learn abstract visual concepts from GCD. This allows us to learn the text-modified image features, which combine complementary and redundant information from the multi-modal data streams into a single, meaningful feature. Secondly, the captions for the GCD dataset are computed with the pretrained CLIP-based image captioning model. Finally, we also demonstrate that the proposed the dataset is also useful for generating greeting card images using pre-trained text-to-image generation model.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningImage GenerationText to Image GenerationText-to-Image GenerationSimilar Papers 제목 키워드 기반
MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving
Instance segmentation is a fundamental research in computer vision, especially in autonomous driving. However, manual mask annotation for instance segmentation is quite time-consuming and costly. To address this problem,…
3D Instance SegmentationAutonomous DrivingInstance SegmentationSegmentation+2CIEC: Coupling Implicit and Explicit Cues for Multimodal Weakly Supervised Manipulation Localization
To mitigate the threat of misinformation, multimodal manipulation localization has garnered growing attention. Consider that current methods rely on costly and time-consuming fine-grained annotations, such as patch/token…
Prompt learning with bounding box constraints for medical image segmentation
Pixel-wise annotations are notoriously labourious and costly to obtain in the medical domain. To mitigate this burden, weakly supervised approaches based on bounding box annotations-much easier to acquire-offer a practic…
Medical Image SegmentationMulti-scale Bottleneck Transformer for Weakly Supervised Multimodal Violence Detection
Weakly supervised multimodal violence detection aims to learn a violence detection model by leveraging multiple modalities such as RGB, optical flow, and audio, while only video-level annotations are available. In the pu…
Anomaly Detection In Surveillance VideosOptical Flow EstimationVoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, most existing end-to-end VLP methods use high-resolution image-text box data to p…
object-detectionObject DetectionReferring ExpressionReferring Expression Comprehension