Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs
We introduce Cube Bench, a Rubik's-cube benchmark for evaluating spatial and sequential reasoning in multimodal large language models (MLLMs). The benchmark decomposes performance into five skills: (i) reconstructing cube faces from images and text, (ii) choosing the optimal next move, (iii) predicting the outcome of a candidate move without applying it, (iv) executing multi-step plans while recovering from mistakes, and (v) detecting and revising one's own errors. Using a shared set of scrambled cube states, identical prompts and parsers, and a single distance-to-solved metric, we compare recent MLLMs side by side as a function of scramble depth. Across seven MLLMs, accuracy drops sharply with depth; once a trajectory stalls or diverges, models rarely recover, and high face-reconstruction accuracy does not guarantee competent action selection or multi-step execution. A pronounced closed- vs open-source gap emerges: the strongest closed model leads on both single-step perception tasks and multi-step control tasks, while open-weight models cluster near chance on the hardest settings; yet even the best MLLM degrades at higher cube complexity. A simple self-correction via reflective thinking yields modest gains but can also introduce overthinking. Cube Bench offers a compact, reproducible probe of sequential spatial reasoning in MLLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Spatial ReasoningVisual ReasoningSimilar Papers 제목 키워드 기반
Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
Spatial cognition is essential for human intelligence, enabling problem-solving through visual simulations rather than solely relying on verbal reasoning. However, existing AI benchmarks primarily assess verbal reasoning…
4kSpatial ReasoningCubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observations
Large Language Model (LLM) agents, while proficient in the digital realm, face a significant gap in physical-world deployment due to the challenge of forming and maintaining a robust spatial mental model. We identify thr…
Spatial ReasoningpySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning
Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To addre…
Spatial Reasoning3D ReconstructionCode GenerationCityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments
Cross-view spatial reasoning is essential for embodied AI, underpinning spatial understanding, mental simulation and planning in complex environments. Existing benchmarks primarily emphasize indoor or street settings, ov…
Spatial ReasoningAuto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models
Modern Vision-Language Models (VLMs) exhibit a critical flaw in compositional reasoning, often confusing "a red cube and a blue sphere" with "a blue cube and a red sphere". Disentangling the visual and linguistic roots o…
Spatial Reasoning