DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

2026-04-10 · Young-Suk Lee, Ramon Fernandez Astudillo, Radu Florian arxiv

Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity (84% excluding stale data), with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity.

📄 PDF Abstract BibTeX arXiv:2604.09251

Code (0)

등록된 구현이 없습니다.

Debate to Align: Reliable Entity Alignment through Two-Stage Multi-Agent Debate

2026-04-15 · Cunda Wang, Ziying Ma, Po Hu, Weihua Wang 외 arxiv

Entity alignment (EA) aims to identify entities referring to the same real-world object across different knowledge graphs (KGs). Recent approaches based on large language models (LLMs) typically obtain entity embeddings …

Representation Learning Knowledge Graphs Entity Alignment Decision Making

Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms

2026-05-28 · Botao Amber Hu, Helena Rong, Max Van Kleek arxiv

As autonomous language model agents proliferate, forming an emerging agentic web with real-world consequences, what credibility signals can you use to decide whether to trust an unfamiliar agent in the wild and delegate …

Adversarial Attack

Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use

2026-04-07 · Wuyang Zhang, Shichao Pei arxiv

Tool-use large language model (LLM) agents are increasingly deployed to support sensitive workflows, relying on tool calls for retrieval, external API access, and session memory management. While prior research has exami…

BYOKG-RAG: Multi-Strategy Graph Retrieval for Knowledge Graph Question Answering

2025-07-05 · Costas Mavromatis, Soji Adeshina, Vassilis N. Ioannidis, Zhen Han 외 arxiv

Knowledge graph question answering (KGQA) presents significant challenges due to the structural and semantic variations across input graphs. Existing works rely on Large Language Model (LLM) agents for graph traversal an…

Graph Question Answering Answer Generation Entity Linking

An Entity Linking Agent for Question Answering

2025-08-05 · Yajie Luo, Yihong Wu, Muzhi Li, Jia Ao Sun 외 arxiv

Some Question Answering (QA) systems rely on knowledge bases (KBs) to provide accurate answers. Entity Linking (EL) plays a critical role in linking natural language mentions to KB entries. However, most existing EL meth…

Question Answering Entity Linking