Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models

2026-01-19 · Felix Mächtle, Jan-Niclas Serr, Nils Loose, Thomas Eisenbarth arxiv

Large Language Models (LLMs) are increasingly integrated into software engineering workflows, yet current benchmarks provide only coarse performance summaries that obscure the diverse capabilities and limitations of these models. This paper investigates whether LLMs' code-comprehension performance aligns with traditional human-centric software metrics or instead reflects distinct, non-human regularities. We introduce a diagnostic framework that reframes code understanding as a binary input-output consistency task, enabling the evaluation of classification and generative models. Using a large-scale dataset, we correlate model performance with traditional, human-centric complexity metrics, such as lexical size, control-flow complexity, and abstract syntax tree structure. Our analyses reveal minimal correlation between human-defined metrics and LLM success (AUROC 0.63), while shadow models achieve substantially higher predictive performance (AUROC 0.86), capturing complex, partially predictable patterns beyond traditional software measures. These findings suggest that LLM comprehension reflects model-specific regularities only partially accessible through either human-designed or learned features, emphasizing the need for benchmark methodologies that move beyond aggregate accuracy and toward instance-level diagnostics, while acknowledging fundamental limits in predicting correct outcomes.

📄 PDF Abstract BibTeX arXiv:2601.12951

Code (0)

등록된 구현이 없습니다.

A Code Comprehension Benchmark for Large Language Models for Code

2025-07-14 · Jayant Havare, Saurav Chaudhary, Ganesh Ramakrishnan, Kaushik Maharajan 외 arxiv

Large Language Models have shown impressive capabilities in coding tasks like code generation and code completion, as they have been trained on a large amount of code data. Also, since one of the core pretraining objecti…

Code Generation Code Completion

The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs

2025-12-17 · Tejas Anvekar, Fenil Bardoliya, Pavan K. Turaga, Chitta Baral 외 arxiv

Recent advances in multimodal large language models (MLLMs) have yielded increasingly powerful models, yet their perceptual capacities remain poorly characterized. In practice, most model families scale language componen…

Visual Grounding Image Matching

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

2024-03-23 · HAZ Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad 외

The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visua…

Common Sense Reasoning In-Context Learning Multiple-choice Object Localization+2

BloomVQA: Assessing Hierarchical Multi-modal Comprehension

2023-12-20 · Yunye Gong, Robik Shrestha, Jared Claypoole, Michael Cogswell 외

We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple re…

Data Augmentation Memorization Multiple-choice Visual Question Answering (VQA)

Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering

2025-10-21 · Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata arxiv

This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA. Our comprehensive approach considers a range of settings base…

Question Answering