How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval
Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5% of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Art AnalysisRetrievalSimilar Papers 제목 키워드 기반
Understanding Art through Multi-Modal Retrieval in Paintings
In computer vision, visual arts are often studied from a purely aesthetics perspective, mostly by analysing the visual appearance of an artistic reproduction to infer its style, its author, or its representative features…
Art AnalysisRetrievalA Novel Approach to Artistic Textual Visualization via GAN
While the visualization of statistical data tends to a mature technology, the visualization of textual data is still in its infancy, especially for the artistic text. Due to the fact that visualization of artistic text i…
Generative Adversarial NetworkSemantic Segmentation in Art Paintings
Semantic segmentation is a difficult task even when trained in a supervised manner on photographs. In this paper, we tackle the problem of semantic segmentation of artistic paintings, an even more challenging task becaus…
Domain AdaptationSegmentationSemantic SegmentationStyle TransferCreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion
Although remarkable progress has been made in image style transfer, style is just one of the components of artistic paintings. Directly transferring extracted style features to natural images often results in outputs wit…
Image GenerationStyle TransferText to Image GenerationText-to-Image GenerationA Multi-modal Registration and Visualization Software Tool for Artworks using CraquelureNet
For art investigations of paintings, multiple imaging technologies, such as visual light photography, infrared reflectography, ultraviolet fluorescence photography, and x-radiography are often used. For a pixel-wise comp…