Image Semantic Relation Generation
Scene graphs provide structured semantic understanding beyond images. For downstream tasks, such as image retrieval, visual question answering, visual relationship detection, and even autonomous vehicle technology, scene graphs can not only distil complex image information but also correct the bias of visual models using semantic-level relations, which has broad application prospects. However, the heavy labour cost of constructing graph annotations may hinder the application of PSG in practical scenarios. Inspired by the observation that people usually identify the subject and object first and then determine the relationship between them, we proposed to decouple the scene graphs generation task into two sub-tasks: 1) an image segmentation task to pick up the qualified objects. 2) a restricted auto-regressive text generation task to generate the relation between given objects. Therefore, in this work, we introduce image semantic relation generation (ISRG), a simple but effective image-to-text model, which achieved 31 points on the OpenPSG dataset and outperforms strong baselines respectively by 16 points (ResNet-50) and 5 points (CLIP).
Code (0)
등록된 구현이 없습니다.
Tasks
Image RetrievalImage SegmentationImage to textQuestion AnsweringRelationRelationship DetectionRetrievalSemantic SegmentationText GenerationVisual Question AnsweringVisual Question Answering (VQA)Visual Relationship DetectionSimilar Papers 제목 키워드 기반
LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts
Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis. Prior works mostly rely on pre-trained linguistic models, but a text is often too abstract to properly sp…
Image GenerationLayout-to-Image GenerationObjectScene GenerationGloTok: Global Perspective Tokenizer for Image Reconstruction and Generation
Existing state-of-the-art image tokenization methods leverage diverse semantic features from pre-trained vision models for additional supervision, to expand the distribution of latent representations and thereby improve …
Image ReconstructionBuilding Joint Relationship Attention Network for Image-Text Generation
Attention based methods for image-text generation often focus on visual features individually, while ignoring relationship information among image features that provides important guidance for generating sentences. To al…
Text GenerationObject-Centric Relational Representations for Image Generation
Conditioning image generation on specific features of the desired output is a key ingredient of modern generative models. However, existing approaches lack a general and unified way of representing structural and semanti…
Image GenerationInductive BiasObjectDSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation
Scene graph generation aims to capture detailed spatial and semantic relationships between objects in an image, which is challenging due to incomplete labelling, long-tailed relationship categories, and relational semant…
Graph GenerationGraph MatchingPanoptic Scene Graph GenerationRelation+2