ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation

2026-02-24 · Nabin Oli arxiv

Traditional benchmarks like HumanEval and MBPP test logic and syntax effectively, but fail when code must produce dynamic, pedagogical visuals. We introduce ManiBench, a specialized benchmark evaluating LLM performance in generating Manim CE code, where temporal fidelity and version-aware API correctness are critical. ManiBench targets two key failure modes: Syntactic Hallucinations (valid Python referencing non-existent or deprecated Manim APIs) and Visual-Logic Drift (generated visuals diverging from intended mathematical logic through timing errors or missing causal relationships). The benchmark comprises 150-200 problems across five difficulty levels spanning calculus, linear algebra, probability, topology, and AI, grounded in analysis of 3Blue1Brown's ManimGL source (53,000 lines, 143 scene classes). Evaluation uses a four-tier framework measuring Executability, Version-Conflict Error Rate, Alignment Score, and Coverage Score. An open-source framework automates evaluation across multiple models and prompting strategies. Code, data and benchmark suite are available at https://github.com/nabin2004/ManiBench. and the dataset is hosted on https://huggingface.co/datasets/nabin2004/ManiBench.

📄 PDF Abstract BibTeX arXiv:2603.13251

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation

HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

2025-05-16 · Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani 외

Large multimodal models (LMMs) now excel on many vision language benchmarks, however, they still struggle with human centered criteria such as fairness, ethics, empathy, and inclusivity, key to aligning with human values…

Benchmarking Ethics Fairness Question Answering+3

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

2026-02-05 · Wenbo Wang, Fangyun Wei, QiXiu Li, Xi Chen 외 arxiv

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first f…

Reinforcement Learning

Request-and-Reverify: Hierarchical Hypothesis Testing for Concept Drift Detection with Expensive Labels

2018-06-25 · Shujian Yu, Xiaoyang Wang, Jose C. Principe

One important assumption underlying common classification models is the stationarity of the data. However, in real-world streaming applications, the data concept indicated by the joint distribution of feature and label i…

Attribute Drift Detection General Classification Two-sample testing

DRIFT: Deep Reinforcement Learning for Functional Software Testing

2020-07-16 · Luke Harries, Rebekah Storan Clarke, Timothy Chapman, Swamy V. P. L. N. Nallamalli 외

Efficient software testing is essential for productive software development and reliable user experiences. As human testing is inefficient and expensive, automated software testing is needed. In this work, we propose a R…

Deep Reinforcement Learning Graph Neural Network Q-Learning reinforcement-learning+3

DriftScript: A Domain-Specific Language for Programming Non-Axiomatic Reasoning Agents

2026-03-29 · Seamus Brady arxiv

Non-Axiomatic Reasoning Systems (NARS) provide a framework for building adaptive agents that operate under insufficient knowledge and resources. However, the standard input language, Narsese, poses a usability barrier: i…