MetaCLUE: Towards Comprehensive Visual Metaphors Research
Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationQuestion AnsweringRetrievalSimilar Papers 제목 키워드 기반
Unveiling the Invisible: Captioning Videos with Metaphors
Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent…
MetaClimage: A novel database of visual metaphors related to Climate Change, with costs and benefits analysis
Visual metaphors of climate change (e.g., melting glaciers depicted as a melting ice grenade) are regarded as valuable tools for addressing the complexity of environmental challenges. However, few studies have examined t…
I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors
Visual metaphors are powerful rhetorical devices used to persuade or communicate creative ideas through images. Similar to linguistic metaphors, they convey meaning implicitly through symbolism and juxtaposition of the s…
Visual EntailmentQuantum Machine Learning Playground
This article introduces an innovative interactive visualization tool designed to demystify quantum machine learning (QML) algorithms. Our work is inspired by the success of classical machine learning visualization tools,…
Quantum Machine LearningSky + Fire = Sunset. Exploring Parallels between Visually Grounded Metaphors and Image Classifiers
This work explores the differences and similarities between neural image classifiers{'} mis-categorisations and visually grounded metaphors - that we could conceive as intentional mis-categorisations. We discuss the poss…