From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis
We explore multi-step reasoning in vision-language models (VLMs). The problem is challenging, as reasoning data consisting of multiple steps of visual and language processing are barely available. To overcome the challenge, we first introduce a least-to-most visual reasoning paradigm, which interleaves steps of decomposing a question into sub-questions and invoking external tools for resolving sub-questions. Based on the paradigm, we further propose a novel data synthesis approach that can automatically create questions and multi-step reasoning paths for an image in a bottom-up manner. Our approach divides the complex synthesis task into a few simple sub-tasks, and (almost entirely) relies on open-sourced models to accomplish the sub-tasks. Therefore, the entire synthesis process is reproducible and cost-efficient, and the synthesized data is quality guaranteed. With the approach, we construct $50$k visual reasoning examples. Then, we develop a visual reasoner through supervised fine-tuning, which is capable of generally enhancing the reasoning abilities of a wide range of existing VLMs in a plug-and-play fashion. Extensive experiments indicate that the visual reasoner can consistently and significantly improve four VLMs on four VQA benchmarks. Our code and dataset are available at https://github.com/steven-ccq/VisualReasoner.
Code (2)
Tasks
Visual Question Answering (VQA)Visual ReasoningSimilar Papers 제목 키워드 기반
FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models
Visual Autoregressive (VAR) modeling departs from the next-token prediction paradigm of traditional Autoregressive (AR) models through next-scale prediction, enabling high-quality image generation. However, the VAR parad…
Image GenerationIntegrating Reweighted Least Squares with Plug-and-Play Diffusion Priors for Noisy Image Restoration
Existing plug-and-play image restoration methods typically employ off-the-shelf Gaussian denoisers as proximal operators within classical optimization frameworks based on variable splitting. Recently, denoisers induced b…
Image RestorationLast But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning, but long visual contexts enlarge the KV cache and increase decoding latency. Existing compression methods rely on observation window atte…
Occupant Plugload Management for Demand Response in Commercial Buildings: Field Experimentation and Statistical Characterization
Commercial buildings account for approximately 35% of total US electricity consumption, of which nearly two-thirds is met by fossil fuels resulting in an adverse impact on the environment. This adverse impact can be miti…
ManagementConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
Diffusion transformers have demonstrated strong capabilities in generating high-quality images. However, as model size increases, the growing memory footprint and inference latency pose significant challenges for practic…