Self-Supervised Image-to-Text and Text-to-Image Synthesis
A comprehensive understanding of vision and language and their interrelation are crucial to realize the underlying similarities and differences between these modalities and to learn more generalized, meaningful representations. In recent years, most of the works related to Text-to-Image synthesis and Image-to-Text generation, focused on supervised generative deep architectures to solve the problems, where very little interest was placed on learning the similarities between the embedding spaces across modalities. In this paper, we propose a novel self-supervised deep learning based approach towards learning the cross-modal embedding spaces; for both image to text and text to image generations. In our approach, we first obtain dense vector representations of images using StackGAN-based autoencoder model and also dense vector representations on sentence-level utilizing LSTM based text-autoencoder; then we study the mapping from embedding space of one modality to embedding space of the other modality utilizing GAN and maximum mean discrepancy based generative networks. We, also demonstrate that our model learns to generate textual description from image data as well as images from textual data both qualitatively and quantitatively.
Code (1)
Tasks
Image GenerationImage to textSentenceText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised Learning of Pretext-Invariant Representations
The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many …
Contrastive Learningobject-detectionObject DetectionRepresentation Learning+3Self-Supervised Representation Learning on Document Images
This work analyses the impact of self-supervised pre-training on document images in the context of document image classification. While previous approaches explore the effect of self-supervision on natural images, we sho…
Classificationdocument-image-classificationDocument Image ClassificationGeneral Classification+3DiT: Self-supervised Pre-training for Document Image Transformer
Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we prop…
Document AIdocument-image-classificationDocument Image ClassificationDocument Layout Analysis+5Self-supervised Character-to-Character Distillation for Text Recognition
When handling complicated text images (e.g., irregular structures, low resolution, heavy occlusion, and uneven illumination), existing supervised text recognition methods are data-hungry. Although these methods employ la…
Data AugmentationRepresentation LearningScene Text RecognitionSelf-Learning+5Mixture of Self-Supervised Learning
Self-supervised learning is popular method because of its ability to learn features in images without using its labels and is able to overcome limited labeled datasets used in supervised learning. Self-supervised learnin…
Classificationimage-classificationImage ClassificationSelf-Supervised Learning