Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework
We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs
We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation produced by state-of-the-art multilingual large language models (LLMs). Existing MT benchmarks emphasise toke…
Machine Translation"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?
Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative…
Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!
Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalitie…
Idiom Savant at Semeval-2017 Task 7: Detection and Interpretation of English Puns
This paper describes our system, entitled Idiom Savant, for the 7th Task of the Semeval 2017 workshop, {``}Detection and interpretation of English Puns{''}. Our system consists of two probabilistic models for each type o…
Task 2Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement
Recent advances in Multimodal Large Language Models (MLLMs) have enabled automated generation of structured layouts from natural language descriptions. Existing methods typically follow a code-only paradigm that generate…
Reinforcement Learning