Evaluation of Causal Reasoning for Large Language Models in Contextualized Clinical Scenarios of Laboratory Test Interpretation

2025-09-19 · Balu Bhasuran, Mattia Prosperi, Karim Hanna, John Petrilli, Caretia JeLayne Washington, Zhe He arxiv

This study evaluates causal reasoning in large language models (LLMs) using 99 clinically grounded laboratory test scenarios aligned with Pearl's Ladder of Causation: association, intervention, and counterfactual reasoning. We examined common laboratory tests such as hemoglobin A1c, creatinine, and vitamin D, and paired them with relevant causal factors including age, gender, obesity, and smoking. Two LLMs - GPT-o1 and Llama-3.2-8b-instruct - were tested, with responses evaluated by four medically trained human experts. GPT-o1 demonstrated stronger discriminative performance (AUROC overall = 0.80 +/- 0.12) compared to Llama-3.2-8b-instruct (0.73 +/- 0.15), with higher scores across association (0.75 vs 0.72), intervention (0.84 vs 0.70), and counterfactual reasoning (0.84 vs 0.69). Sensitivity (0.90 vs 0.84) and specificity (0.93 vs 0.80) were also greater for GPT-o1, with reasoning ratings showing similar trends. Both models performed best on intervention questions and worst on counterfactuals, particularly in altered outcome scenarios. These findings suggest GPT-o1 provides more consistent causal reasoning, but refinement is required before adoption in high-stakes clinical applications.

📄 PDF Abstract BibTeX arXiv:2509.16372

Code (0)

등록된 구현이 없습니다.

COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective

2023-05-09 · Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang 외

Detecting commonsense causal relations (causation) between events has long been an essential yet challenging task. Given that events are complicated, an event may have different causes under various contexts. Thus, explo…

Causal Inference CoLA Commonsense Causal Reasoning

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

2026-05-05 · Zhi Xu, Yun Fu arxiv

Causal reasoning in natural language requires identifying relevant variables, understanding their interactions, and reasoning about effects and interventions, often under noisy or ambiguous conditions. While large langua…

COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completion

2021-06-04 · ACL 2021 5 · Debjit Paul, Anette Frank

Despite recent successes of large pre-trained language models in solving reasoning tasks, their inference capabilities remain opaque. We posit that such models can be made more interpretable by explicitly generating inte…

Sentence Story Completion

Visual Abductive Reasoning

2022-03-26 · CVPR 2022 1 · Chen Liang, Wenguan Wang, Tianfei Zhou, Yi Yang

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this pape…

Benchmarking Sentence Visual Abductive Reasoning

Grounding Before Generalizing: How AI Differs from Humans in Causal Transfer

2026-04-27 · Liangru Xiang, Yuxi Ma, Zhihao Cao, Yixin Zhu 외 arxiv

Extracting abstract causal structures and applying them to novel situations is a hallmark of human intelligence. While Large Language Models (LLMs) and Vision Language Models (VLMs) have shown strong performance on a wid…

Reinforcement Learning Multimodal Reasoning