Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering
Radiology visual question answering (RVQA) provides precise answers to questions about chest X-ray images, alleviating radiologists' workload. While recent methods based on multimodal large language models (MLLMs) and re…
Visual Question AnsweringMultimodal ReasoningVernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoning
Instruction-tuned language models can answer the same causal-reasoning question differently after its English variable names are replaced by type-preserving placeholders, although the structural causal model and the gold…
Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models
Recent advances in large language models (LLMs) have demonstrated that reinforcement learning with verifiable rewards (RLVR) can significantly enhance reasoning abilities by directly optimizing correctness, rather than r…
Reinforcement LearningHow Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggre…
Visual Question AnsweringAttribute ExtractionVisual GroundingCAPO: Reinforcing Consistent Reasoning in Medical Decision-Making
In medical visual question answering (Med-VQA), achieving accurate responses relies on three critical steps: precise perception of medical imaging data, logical reasoning grounded in visual input and textual questions, a…
Answer GenerationDecision MakingLogical ReasoningMedical Visual Question Answering+4