Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs. These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems. We introduce blind-spots-bench, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples. We further develop an automated grading pipeline to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models. Our analysis on blind-spots-bench reveals that closed-source frontier models can substantially outperform open-weight models with even approx10% gap, even when they attain comparable performance on existing benchmarks. A more fine-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models. These results highlight the value of blind-spots-bench as a diagnostic stress test for identifying concrete weaknesses in current modern models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Discovering Blind Spots in Reinforcement Learning
Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict the…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Towards a More Rigorous Science of Blindspot Discovery in Image Classification Models
A growing body of work studies Blindspot Discovery Methods ("BDM"s): methods that use an image embedding to find semantically meaningful (i.e., united by a human-understandable concept) subsets of the data where an image…
Dimensionality Reductionimage-classificationImage ClassificationBlind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots
Real noisy-clean pairs on a large scale are costly and difficult to obtain. Meanwhile, supervised denoisers trained on synthetic data perform poorly in practice. Self-supervised denoisers, which learn only from single no…
DenoisingImage DenoisingMethodological Blind Spots in Machine Learning Fairness: Lessons from the Philosophy of Science and Computer Science
In the ML fairness literature, there have been few investigations through the viewpoint of philosophy, a lens that encourages the critical evaluation of basic assumptions. The purpose of this paper is to use three ideas …
BIG-bench Machine LearningFairnessPhilosophyAssessment of Vehicular Vision Obstruction Due to Driver-Side B-Pillar and Remediation with Blind Spot Eliminator
Blind spots created by the driver-side B-pillar impair the ability of the driver to assess their surroundings accurately, significantly contributing to the frequency and severity of vehicular accidents. Vehicle manufactu…