Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?
Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study answers an unexplored question about the impact of different question types on LLM accuracy on reasoning tasks. We investigate the performance of five LLMs on three different types of questions using quantitative and deductive reasoning tasks. The performance metrics include accuracy in the reasoning steps and choosing the final answer. Key Findings: (1) Significant differences exist in LLM performance across different question types. (2) Reasoning accuracy does not necessarily correlate with the final selection accuracy. (3) The number of options and the choice of words, influence LLM performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models
The burgeoning interest in Multimodal Large Language Models (MLLMs), such as OpenAI's GPT-4V(ision), has significantly impacted both academic and industrial realms. These models enhance Large Language Models (LLMs) with …
HellaSwagEvaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
With the rapid advancement of Artificial Intelligence (AI), Large Language Models (LLMs) have significantly impacted a wide array of domains, including healthcare, engineering, science, education, and mathematical reason…
Formal LogicMathematical ReasoningMMLUWhen Do Program-of-Thoughts Work for Reasoning?
In the realm of embodied artificial intelligence, the reasoning capabilities of Large Language Models (LLMs) play a pivotal role. Although there are effective methods like program-of-thought prompting for LLMs which uses…
Code GenerationMathematical ReasoningZero-Shot Multi-Hop Question Answering via Monte-Carlo Tree Search with Large Language Models
Recent advances in large language models (LLMs) have significantly impacted the domain of multi-hop question answering (MHQA), where systems are required to aggregate information and infer answers from disparate pieces o…
Multi-hop Question AnsweringQuestion AnsweringHow Does Quantization Affect Multilingual LLMs?
Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantization on LLMs in English, none have evaluated across lan…
Mathematical ReasoningQuantization