GAIA: a benchmark for General AI Assistants

2023-11-21 · Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann Lecun, Thomas Scialom

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

📄 PDF Abstract BibTeX arXiv:2311.12983

Code (2)

metr/public-tasks pytorch

princeton-pli/hal-harness

Tasks

Philosophy

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding

2026-08-12 · Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu 외 hf

Video understanding is a fundamental task for evaluating the capabilities of multimodal large language models (MLLMs). However, existing leading models have already achieved approximately 90% accuracy on the Video-MME le…

Video Question Answering

OmniGAIA: Towards Native Omni-Modal AI Agents

2026-02-26 · Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li 외 arxiv

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily …

Affordable AI Assistants with Knowledge Graph of Thoughts

2025-04-03 · Maciej Besta, Lorenzo Paleari, Jia Hao Andrea Jiang, Robert Gerstenberger 외

Large Language Models (LLMs) are revolutionizing the development of AI assistants capable of performing diverse tasks across domains. However, current state-of-the-art LLM-driven agents face significant challenges, inclu…

Knowledge Graphs LLM real-life tasks Multimodal Reasoning RAG

JoyAgent-JDGenie: Technical Report on the GAIA

2025-10-01 · Jiarun Liu, Shiyue Xu, Shangkun Liu, Yang Li 외 arxiv

Large Language Models are increasingly deployed as autonomous agents for complex real-world tasks, yet existing systems often focus on isolated improvements without a unifying design for robustness and adaptability. We p…

OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

2024-02-12 · Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng 외

Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. …