paper-with-me

홈 › Papers

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

2026-08-19 · Zinuo Guo, Min Zhang, Bo Jiang arxiv

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

📄 PDF Abstract BibTeX arXiv:2608.18586

Code (0)

등록된 구현이 없습니다.

Tasks

Handwritten Text Recognition

Similar Papers 제목 키워드 기반

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

2026-07-28 · Rui Yang, Weihao Xuan, Yi Lin, Zhuhan Bao 외 arxiv

Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic…

MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

2026-02-20 · Ziqiao Shang, Lingyue Ge, Zi-Jian Cheng, Shi-Yu Tian 외 arxiv

Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI). Yet existing benchmarks remain inadequate for rigorously measuring their reasoning capa…

Multimodal Reasoning

SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models

2026-01-11 · Yuhang Su, Mei Wang, Yaoyao Zhong, Guozhang Li 외 arxiv

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limita…

MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark

2026-04-12 · Junzhi Ning, Jiashi Lin, Yingying Fang, Wei Li 외 arxiv

Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinica…

Clinical Knowledge

AEGIS: A Holistic Benchmark for Evaluating Forensic Analysis of AI-Generated Academic Images

2026-04-30 · Bo Zhang, Tzu-Yen Ma, Zichen Tang, Junpeng Ding 외 arxiv

We introduce AEGIS, A holistic benchmark for Evaluating forensic analysis of AI-Generated academic ImageS. Compared to existing benchmarks, AEGIS features three key advances: (1) Domain-Specific Complexity: covering seve…