Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
This paper proposes a novel framework for generating lingual descriptions of indoor scenes. Whereas substantial efforts have been made to tackle this problem, previous approaches focusing primarily on generating a single sentence for each image, which is not sufficient for describing complex scenes. We attempt to go beyond this, by generating coherent descriptions with multiple sentences. Our approach is distinguished from conventional ones in several aspects: (1) a 3D visual parsing system that jointly infers objects, attributes, and relations; (2) a generative grammar learned automatically from training text; and (3) a text generation algorithm that takes into account the coherence among sentences. Experiments on the augmented NYU-v2 dataset show that our framework can generate natural descriptions with substantially higher ROGUE scores compared to those produced by the baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceText GenerationSimilar Papers 제목 키워드 기반
Less is More: Generating Grounded Navigation Instructions from Landmarks
We study the automatic generation of navigation instructions from 360-degree images captured on indoor routes. Existing generators suffer from poor visual grounding, causing them to rely on language priors and hallucinat…
DecoderInstruction FollowingVisual GroundingGenerating Image Descriptions using Multilingual Data
On generating coherent multilingual descriptions of museum objects from Semantic Web ontologies
Image Pivoting for Learning Multilingual Multimodal Representations
In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image unders…
Image DescriptionImage RetrievalSemantic Textual SimilaritySentence-Level Multilingual Multi-modal Embedding for Natural Language Processing
We propose a novel discriminative ranking model that learns embeddings from multilingual and multi-modal data, meaning that our model can take advantage of images and descriptions in multiple languages to improve embeddi…
Machine TranslationNMTRe-RankingSemantic Textual Similarity+3