FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference. They do not assess whether models can reason over the global structural properties of formal diagrams, such as topology, conservation constraints, and the consistent mapping between visual patterns and algebraic expressions. We introduce FeynmanBench, a benchmark of over 2,000 tasks centered on Feynman diagrams spanning the electromagnetic, weak, and strong interactions of the Standard Model. Each instance couples a diagram image with minimal textual conventions and requires models to recover the full physical content -- vertex inventory, propagator types, topological connectivity, momentum routing, and the complete scattering amplitude. An automated generation and verification pipeline produces the diagrams, annotations, and reference answers under standardized rules. Evaluating 19 state-of-the-art multimodal LLMs, we find a consistent failure pattern: models achieve 70--95\% on local recognition (vertex and propagator identification) but collapse to 13--17\% on topological reconstruction (CP3), and near zero on full algebraic derivation (CP5). FeynmanBench offers a controlled testbed for multimodal reasoning over formal scientific diagrams and highlights fundamental limitations of current architectures in topology-sensitive scientific reasoning.
Code (0)
등록된 구현이 없습니다.
Tasks
Information ExtractionMultimodal ReasoningSimilar Papers 제목 키워드 기반
Introducing the diagrammatic semiotic mode
As the use and diversity of diagrams across many disciplines grows, there is an increasing interest in the diagrams research community concerning how such diversity might be documented and explained. In this article, we …
DiversityGeneral ClassificationPi-GPS: Enhancing Geometry Problem Solving by Unleashing the Power of Diagrammatic Information
Geometry problem solving has garnered increasing attention due to its potential applications in intelligent education field. Inspired by the observation that text often introduces ambiguities that diagrams can clarify, t…
Geometry Problem SolvingMathematical ReasoningBenchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving
Retrieval-augmented generation (RAG) with foundation models has achieved strong performance across diverse tasks, but their capacity for expert-level reasoning-such as solving Olympiad-level physics problems-remains larg…
Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
Multimodal models, such as the Contrastive Language-Image Pre-training (CLIP) model, have demonstrated remarkable success in aligning visual and linguistic representations. However, these models exhibit limitations when …
Visual Question AnsweringContrastive LearningImage-text matchingChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult. Existing benchmarks focus on narrow chart types and leave diagramma…