Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric

2026-05-07 · Ying Gu, Mei Chee Leong, Hui Li Tan, Shangbo Mao, Liyuan Li, Nancy Chen arxiv

Dominant accuracy evaluation might reward unwarranted guessing of Large Language Models, and it might not be applicable to novel tasks for model validation without ground-truth (gt) annotation. Based on basic logic principle, we propose a novel framework to evaluate the vision-language logical consistency of MLLMs on both sufficient and necessary cause-effect relations. We define Vision-Language Logical Consistency Metric (VL-LCM) on traditional MC-VQA tests, and recent NaturalBench tests without the need for gt annotation. Through systematic experiments on representative VL benchmark MMMU and recent VL challenges like NaturalBench, we evaluated 11 recent open-source MLLMs from 4 frontier families. Our findings reveal that, despite significant progress of recent MLLMs on accuracy, logical consistency lags behind significantly. Extensive evaluations on the correlations of VL-LCM with metrics on gt, the reliability of LCM, and the relation of VL-LCM with response distribution justify the validity and applicability of VL-LCM even without gt annotation. Our findings suggest that, beyond accuracy, logical consistency could be employed for both accuracy and reliability. VL-LCM can also be employed for MLLM selection, validation, and reliable answer justification in novel tasks without gt annotation.

📄 PDF Abstract BibTeX arXiv:2605.06201

Code (0)

등록된 구현이 없습니다.

Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks

2026-03-12 · Mei Chee Leong, Ying Gu, Hui Li Tan, Liyuan Li 외 arxiv

Frontier Multimodal Large Language Models (MLLMs) exhibit remarkable capabilities in Visual-Language Comprehension (VLC) tasks. However, they are often deployed as zero-shot solution to new tasks in a black-box manner. V…

Relational Reasoning Logical Reasoning

Low-cost concept-based localized explanations: How far can we get with training-free approaches?

2026-06-27 · Darian Fernández-Gutiérrez, Rafael Bello, Marilyn Bello, Natalia Díaz-Rodríguez arxiv

Concept-based Explainable AI (C-XAI) seeks human-understandable explanations grounded in semantic concepts, yet validation is limited by the scarcity of fine-grained concept annotations. We evaluate whether mid-scale Mul…

HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

2024-06-27 · Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao 외

The rapid development of multimodal large language models (MLLMs), such as GPT-4V, has led to significant advancements. However, these models still face challenges in medical multimodal capabilities due to limitations in…

Visual Question Answering (VQA)

Visual Prompting in Multimodal Large Language Models: A Survey

2024-09-05 · Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li 외

Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-graine…

In-Context Learning Prompt Learning Survey Visual Grounding+1

VOLMO: Versatile and Open Large Models for Ophthalmology

2026-03-25 · Zhenyue Qin, Younjoon Chung, Elijah Lee, Wanyue Feng 외 arxiv

Vision impairment affects millions globally, and early detection is critical to preventing irreversible vision loss. Ophthalmology workflows require clinicians to integrate medical images, structured clinical data, and f…