ViPE: Visualise Pretty-much Everything

2023-10-16 · Hassan Shahmohammadi, Adhiraj Ghosh, Hendrik P. A. Lensch

Figurative and non-literal expressions are profoundly integrated in human communication. Visualising such expressions allow us to convey our creative thoughts, and evoke nuanced emotions. Recent text-to-image models like Stable Diffusion, on the other hand, struggle to depict non-literal expressions. Recent works primarily deal with this issue by compiling humanly annotated datasets on a small scale, which not only demands specialised expertise but also proves highly inefficient. To address this issue, we introduce ViPE: Visualise Pretty-much Everything. ViPE offers a series of lightweight and robust language models that have been trained on a large-scale set of lyrics with noisy visual descriptions that represent their implicit meaning. The synthetic visual descriptions are generated by GPT3.5 relying on neither human annotations nor images. ViPE effectively expresses any arbitrary piece of text into a visualisable description, enabling meaningful and high-quality image generation. We provide compelling evidence that ViPE is more robust than GPT3.5 in synthesising visual elaborations. ViPE also exhibits an understanding of figurative expressions comparable to human experts, providing a powerful and open-source backbone to many downstream applications such as music video and caption generation.

📄 PDF Abstract BibTeX arXiv:2310.10543

Code (2)

hazel1994/vipe 공식 구현 pytorch

Hazel1994/ViPE-Videos pytorch

Tasks

Caption GenerationFigurative Language VisualizationImage Generation

Methods 이 논문이 사용한 방법론

CoT Prompting Chain-of-thought prompts contain a series of intermediate reasoning steps, and they are shown to significantly improve the ability of large language models to perform certain…

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Analyzing Modular Approaches for Visual Question Decomposition

2023-11-10 · Apoorv Khandelwal, Ellie Pavlick, Chen Sun

Modular neural networks without additional training have recently been shown to surpass end-to-end neural networks on challenging vision-language tasks. The latest such methods simultaneously introduce LLM-based code gen…

Code Generation Visual Question Answering (VQA)

MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning

2022-05-25 · Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi 외

Many recent breakthroughs in multi-agent reinforcement learning (MARL) require the use of deep neural networks, which are challenging for human experts to interpret and understand. On the other hand, existing work on int…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ViPE: Video Pose Engine for 3D Geometric Perception

2025-08-12 · Jiahui Huang, Qunjie Zhou, Hesam Rabeti, Aleksandr Korovko 외 arxiv

Accurate 3D geometric perception is an important prerequisite for a wide range of spatial AI systems. While state-of-the-art methods depend on large-scale training data, acquiring consistent and precise 3D annotations fr…

Pose Estimation

ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model

2025-10-28 · Juntian Zhang, Song Jin, Chuanqi Cheng, Yuhan Liu 외 arxiv

The limited capacity for fine-grained visual perception presents a critical bottleneck for Vision-Language Models (VLMs) in real-world applications. Addressing this is challenging due to the scarcity of high-quality data…

Reinforcement Learning

V-VIPE: Variational View Invariant Pose Embedding

2024-07-09 · Mara Levy, Abhinav Shrivastava

Learning to represent three dimensional (3D) human pose given a two dimensional (2D) image of a person, is a challenging problem. In order to make the problem less ambiguous it has become common practice to estimate 3D p…

Decoder Diversity