ViperGPT: Visual Inference via Python Execution for Reasoning

2023-03-14 · ICCV 2023 1 · Dídac Surís, Sachit Menon, Carl Vondrick

Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.

📄 PDF Abstract BibTeX arXiv:2303.08128

Code (1)

cvlab-columbia/viper 공식 구현 pytorch

Tasks

Code Generation Video Question Answering Zero-Shot Video Question Answer

A Fast, Reliable, and Secure Programming Language for LLM Agents with Code Actions

2025-06-13 · Stephen Mell, Botong Zhang, David Mell, Shuo Li 외

Modern large language models (LLMs) are often deployed as agents, calling external tools adaptively to solve tasks. Rather than directly calling tools, it can be more effective for LLMs to write code to perform the tool …

Conformal Prediction Question Answering Uncertainty Quantification Visual Question Answering

Language-Vision Planner and Executor for Text-to-Visual Reasoning

2025-06-09 · Yichang Xu, Gaowen Liu, Ramana Rao Kompella, Sihao Hu 외

The advancement in large language models (LLMs) and large vision models has fueled the rapid progress in multi-modal visual-text reasoning capabilities. However, existing vision-language models (VLMs) to date suffer from…

In-Context Learning MME Task Planning Visual Reasoning

AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering

2024-07-28 · Mahiro Ukai, Shuhei Kurita, Atsushi Hashimoto, Yoshitaka Ushiku 외

Visual question answering aims to provide responses to natural language questions given visual input. Recently, visual programmatic models (VPMs), which generate executable programs to answer questions through large lang…

Question Answering Visual Question Answering

Analyzing Modular Approaches for Visual Question Decomposition

2023-11-10 · Apoorv Khandelwal, Ellie Pavlick, Chen Sun

Modular neural networks without additional training have recently been shown to surpass end-to-end neural networks on challenging vision-language tasks. The latest such methods simultaneously introduce LLM-based code gen…

Code Generation Visual Question Answering (VQA)

PropTest: Automatic Property Testing for Improved Visual Programming

2024-03-25 · Jaywon Koo, Ziyan Yang, Paola Cascante-Bonilla, Baishakhi Ray 외

Visual Programming has recently emerged as an alternative to end-to-end black-box visual reasoning models. This type of method leverages Large Language Models (LLMs) to generate the source code for an executable computer…

Question Answering Referring Expression Referring Expression Comprehension Visual Question Answering+1