Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
Generating semantic layout from scene graph is a crucial intermediate task connecting text to image. We present a conceptually simple, flexible and general framework using sequence to sequence (seq-to-seq) learning for this task. The framework, called Seq-SG2SL, derives sequence proxies for the two modality and a Transformer-based seq-to-seq model learns to transduce one into the other. A scene graph is decomposed into a sequence of semantic fragments (SF), one for each relationship. A semantic layout is represented as the consequence from a series of brick-action code segments (BACS), dictating the position and scale of each object bounding box in the layout. Viewing the two building blocks, SF and BACS, as corresponding terms in two different vocabularies, a seq-to-seq model is fittingly used to translate. A new metric, semantic layout evaluation understudy (SLEU), is devised to evaluate the task of semantic layout prediction inspired by BLEU. SLEU defines relationships within a layout as unigrams and looks at the spatial distribution for n-grams. Unlike the binary precision of BLEU, SLEU allows for some tolerances spatially through thresholding the Jaccard Index and is consequently more adapted to the task. Experimental results on the challenging Visual Genome dataset show improvement over a non-sequential approach based on graph convolution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple st…
Image GenerationHeterogeneous Trajectory Forecasting via Risk and Scene Graph Learning
Heterogeneous trajectory forecasting is critical for intelligent transportation systems, but it is challenging because of the difficulty of modeling the complex interaction relations among the heterogeneous road agents a…
Graph LearningTrajectory ForecastingLayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
Designing realistic multi-object scenes requires not only generating images, but also planning spatial layouts that respect semantic relations and physical plausibility. On one hand, while recent advances in diffusion mo…
Spatial ReasoningImage GenerationLayoutGAN: Generating Graphic Layouts with Wireframe Discriminator
Layouts are important for graphic design and scene generation. We propose a novel generative adversarial network, named as LayoutGAN, that synthesizes graphic layouts by modeling semantic and geometric relations of 2D el…
Generative Adversarial NetworkLayout GenerationScene GenerationScene Graph Disentanglement and Composition for Generalizable Complex Image Generation
There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple object…
DisentanglementImage Generation