DEEPQUESTION: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance
LLMs often excel on standard benchmarks but falter on real-world tasks. We introduce DeepQuestion, a scalable automated framework that augments existing datasets based on Bloom's taxonomy and creates novel questions that trace original solution paths to probe evaluative and creative skills. Extensive experiments across ten open-source and proprietary models, covering both general-purpose and reasoning LLMs, reveal substantial performance drops (even up to 70% accuracy loss) on higher-order tasks, underscoring persistent gaps in deep reasoning. Our work highlights the need for cognitively diverse benchmarks to advance LLM progress. DeepQuestion and related datasets will be released upon acceptance of the paper.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability
Generating long, informative, and factual outputs remains a major challenge for Large Language Models (LLMs). Existing benchmarks for long-form generation typically assess real-world queries with hard-to-verify metrics o…
WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenario
It presents significant challenges to generate comprehensive and accurate Wikipedia articles for newly emerging events under a real-world scenario. Existing attempts fall short either by focusing only on short snippets o…
ArticlesRAGRetrievalConTSG-Bench: A Unified Benchmark for Conditional Time Series Generation
Conditional time series generation plays a critical role in addressing data scarcity and enabling causal analysis in real-world applications. Despite its increasing importance, the field lacks a standardized and systemat…
Machine Learning for Synthetic Data Generation: A Review
Machine learning heavily relies on data, but real-world applications often encounter various data-related issues. These include data of poor quality, insufficient data points leading to under-fitting of machine learning …
FairnessSynthetic Data GenerationTowards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI,…
Autonomous Driving3D Generation