Aesthetically Relevant Image Captioning
Image aesthetic quality assessment (AQA) aims to assign numerical aesthetic ratings to images whilst image aesthetic captioning (IAC) aims to generate textual descriptions of the aesthetic aspects of images. In this paper, we study image AQA and IAC together and present a new IAC method termed Aesthetically Relevant Image Captioning (ARIC). Based on the observation that most textual comments of an image are about objects and their interactions rather than aspects of aesthetics, we first introduce the concept of Aesthetic Relevance Score (ARS) of a sentence and have developed a model to automatically label a sentence with its ARS. We then use the ARS to design the ARIC model which includes an ARS weighted IAC loss function and an ARS based diverse aesthetic caption selector (DACS). We present extensive experimental results to show the soundness of the ARS concept and the effectiveness of the ARIC model by demonstrating that texts with higher ARS's can predict the aesthetic ratings more accurately and that the new ARIC model can generate more accurate, aesthetically more relevant and more diverse image captions. Furthermore, a large new research database containing 510K images with over 5 million comments and 350K aesthetic scores, and code for implementing ARIC are available at https://github.com/PengZai/ARIC.
Code (1)
Tasks
Image CaptioningSentenceSimilar Papers 제목 키워드 기반
On the mathematics of beauty: beautiful images
In this paper, we will study the simplest kind of beauty which can be found in simple visual patterns. The proposed approach shows that aesthetically appealing patterns deliver higher amount of information over multiple …
Repurposing Existing Deep Networks for Caption and Aesthetic-Guided Image Cropping
We propose a novel optimization framework that crops a given image based on user description and aesthetics. Unlike existing image cropping methods, where one typically trains a deep network to regress to crop parameters…
Image CaptioningImage CroppingTemporal Knowledge-Aware Image Captioning
Contextualized image captioning is a task that extends beyond generating a purely visual description of the image content and aims to produce a caption that is influenced by the context and informed by the real world kno…
Caption GenerationImage CaptioningWorld KnowledgeICECAP: Information Concentrated Entity-aware Image Captioning
Most current image captioning systems focus on describing general image content, and lack background knowledge to deeply understand the image, such as exact named entities or concrete events. In this work, we focus on th…
ArticlesImage CaptioningRetrievalSentenceImage Captioning as Neural Machine Translation Task in SOCKEYE
Image captioning is an interdisciplinary research problem that stands between computer vision and natural language processing. The task is to generate a textual description of the content of an image. The typical model u…
DecoderImage CaptioningMachine TranslationSentence+1