SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese
We introduce SuperCLUE-Math6(SC-Math6), a new benchmark dataset to evaluate the mathematical reasoning abilities of Chinese language models. SC-Math6 is designed as an upgraded Chinese version of the GSM8K dataset with enhanced difficulty, diversity, and application scope. It consists of over 2000 mathematical word problems requiring multi-step reasoning and providing natural language solutions. We propose an innovative scheme to quantify the reasoning capability of large models based on performance over problems with different reasoning steps. Experiments on 13 representative Chinese models demonstrate a clear stratification of reasoning levels, with top models like GPT-4 showing superior performance. SC-Math6 fills the gap in Chinese mathematical reasoning benchmarks and provides a comprehensive testbed to advance the intelligence of Chinese language models.
Code (1)
Tasks
DiversityGSM8KMathMathematical ReasoningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have led to strong reasoning ability across a wide range of tasks. However, their ability to perform mathematical reasoning from spoken input re…
BenchmarkingMathMathematical Problem-SolvingMathematical Reasoning+1The mechanization of science illustrated by the Lean formalization of the multi-graded Proj construction
We formalize the multi-graded Proj construction in Lean4, illustrating mechanized mathematics and formalization.
SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications
The SuperCLUE-Fin (SC-Fin) benchmark is a pioneering evaluation framework tailored for Chinese-native financial large language models (FLMs). It assesses FLMs across six financial application domains and twenty-five spec…
Computational EfficiencyLogical ReasoningManagementSuperCLUE: A Comprehensive Chinese Large Language Model Benchmark
Large language models (LLMs) have shown the potential to be integrated into human daily lives. Therefore, user preference is the most critical criterion for assessing LLMs' performance in real-world scenarios. However, e…
Language ModelingLanguage ModellingLarge Language ModelmodelAdaptive Strategy for Resetting a Non-stationary Markov Chain during Learning via Joint Stochastic Approximation
In this paper, we tackle the non-stationary kernel problem of the JSA algorithm by Ou and Song 2020, a recent proposal that learns a deep generative model $p_\theta(\mathbf{x},\mathbf{h})$ and a corresponding approximate…