paper-with-me

홈 › Papers

The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost

2026-04-03 · Scott Frohn arxiv

Strategic model selection and reasoning settings are more effective than ensembling for optimizing automated scoring with large language models (LLMs). We examined self-consistency (intra-model majority voting) and reasoning effort for scoring conversation-based assessment items in high school mathematics, evaluating 900 student conversations against human-scored ground truths using frontier and low-cost models from OpenAI and Google. Temperature sampling significantly improved accuracy over deterministic calls, but increasing ensemble size (j = 1 to 7) produced no significant gains. Higher reasoning effort showed a significant positive linear trend with scoring accuracy, though the benefit varied by model family. An efficiency frontier analysis identified Gemini 3.1 Pro Preview at low reasoning as the most accurate but costly configuration; GPT-5.4 Nano and Mini with no reasoning offered the best cost-performance balance.

📄 PDF Abstract BibTeX arXiv:2604.26954

Code (0)

등록된 구현이 없습니다.

Tasks

High School Mathematics

Similar Papers 제목 키워드 기반

Internal Consistency and Self-Feedback in Large Language Models: A Survey

2024-07-19 · Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang 외

Large language models (LLMs) often exhibit deficient reasoning or generate hallucinations. To address these, studies prefixed with "Self-" such as Self-Consistency, Self-Improve, and Self-Refine have been initiated. They…

Impact of Automatic Segmentation on the Quality, Productivity and Self-reported Post-editing Effort of Intralingual Subtitles

2016-05-01 · LREC 2016 5 · Aitor {\'A}lvarez, Marina Balenciaga, Arantza del Pozo, Haritz Arzelus 외

This paper describes the evaluation methodology followed to measure the impact of using a machine learning algorithm to automatically segment intralingual subtitles. The segmentation quality, productivity and self-report…

BIG-bench Machine LearningSegmentation

Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning

2024-08-24 · Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan 외

Self-consistency (SC), a widely used decoding strategy for chain-of-thought reasoning, shows significant gains across various multi-step reasoning tasks but comes with a high cost due to multiple sampling with the preset…

Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models

2024-04-17 · Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim 외

This paper studies the relationship between the surface form of a mathematical problem and its solvability by large language models. We find that subtle alterations in the surface form can significantly impact the answer…

FormLanguage Model EvaluationLanguage ModelingLanguage Modelling+1

Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging

2024-04-19 · Chia-Hsuan Chang, Mary M. Lucas, Yeawon Lee, Christopher C. Yang 외

Advances in large language models (LLMs) have encouraged their adoption in the healthcare domain where vital clinical information is often contained in unstructured notes. Cancer staging status is available in clinical r…

Language ModelingLanguage ModellingLarge Language Model