CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?
We present the Chinese Elementary School Math Word Problems (CMATH) dataset, comprising 1.7k elementary school-level math word problems with detailed annotations, source from actual Chinese workbooks and exams. This dataset aims to provide a benchmark tool for assessing the following question: to what grade level of elementary school math do the abilities of popular large language models (LLMs) correspond? We evaluate a variety of popular LLMs, including both commercial and open-source options, and discover that only GPT-4 achieves success (accuracy $\geq$ 60\%) across all six elementary school grades, while other models falter at different grade levels. Furthermore, we assess the robustness of several top-performing LLMs by augmenting the original problems in the CMATH dataset with distracting information. Our findings reveal that GPT-4 is able to maintains robustness, while other model fail. We anticipate that our study will expose limitations in LLMs' arithmetic and reasoning capabilities, and promote their ongoing development and advancement.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMathMath Word Problem SolvingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KwaiYiiMath: Technical Report
Recent advancements in large language models (LLMs) have demonstrated remarkable abilities in handling a variety of natural language processing (NLP) downstream tasks, even on mathematical tasks requiring multi-step reas…
Arithmetic ReasoningGSM8KMathematical ReasoningChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questions
Mathematical questioning is crucial for assessing students problem-solving skills. Since manually creating such questions requires substantial effort, automatic methods have been explored. Existing state-of-the-art model…
Arithmetic ReasoningMathQuestion AnsweringDocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents
Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert do…
Math構建一個中文國小數學文字問題語料庫(Building a Corpus for Developing the Chinese Elementary School Math Word Problem Solver)[In Chinese]
Zero-shot Chinese Discourse Dependency Parsing via Cross-lingual Mapping
Due to the absence of labeled data, discourse parsing still remains challenging in some languages. In this paper, we present a simple and efficient method to conduct zero-shot Chinese text-level dependency parsing by lev…
Dependency ParsingDiscourse ParsingSentence