Evaluation Sheet for Deep Research: A Use Case for Academic Survey Writing
Large Language Models (LLMs) powered with argentic capabilities are able to do knowledge-intensive tasks without human involvement. A prime example of this tool is Deep research with the capability to browse the web, extract information and generate multi-page reports. In this work, we introduce an evaluation sheet that can be used for assessing the capability of Deep Research tools. In addition, we selected academic survey writing as a use case task and evaluated output reports based on the evaluation sheet we introduced. Our findings show the need to have carefully crafted evaluation standards. The evaluation done on OpenAI`s Deep Search and Google's Deep Search in generating an academic survey showed the huge gap between search engines and standalone Deep Research tools, the shortcoming in representing the targeted area.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Ethics Sheets for AI Tasks
Several high-profile events, such as the mass testing of emotion recognition systems on vulnerable sub-populations and using question answering systems to make moral judgments, have highlighted how technology will often …
ArticlesEmotion RecognitionEthicsQuestion AnsweringEthics Sheets for AI Tasks
Several high-profile events, such as the mass testing of emotion recognition systems on vulnerable sub-populations and using question answering systems to make moral judgments, have highlighted how technology will often …
ArticlesEmotion RecognitionEthicsQuestion AnsweringDeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys
The rapid development of automated survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surveys. Most existing benchmarks first const…
Leakage and the Reproducibility Crisis in ML-based Science
The use of machine learning (ML) methods for prediction and forecasting has become widespread across the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based s…
SurveyResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents
Large language models (LLMs) have exhibited remarkable performance across various tasks in natural language processing. Nevertheless, challenges still arise when these tasks demand domain-specific expertise and advanced …
BenchmarkingSurvey