OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks

2026-01-28 · Jing Wu, Daphne Barretto, Yiye Chen, Nicholas Gydé, Yanan Jian, Yuhang He, Vibhav Vineet arxiv

Long-horizon, repetitive workflows are common in professional settings, such as processing expense reports from receipts and entering student grades from exam papers. These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process. However, they are ideal for Computer-Use Agents (CUAs) due to their structured, recurring sub-workflows with logic that can be systematically learned. Identifying the absence of an evaluation benchmark as a primary bottleneck, we establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents. We then introduce a cost-effective method to construct a condensed demonstration using only few-shot examples to teach agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections. Extensive experiments demonstrate both the inherent challenges of these tasks and the effectiveness of our proposed method. Project website: https://os-marathon.github.io/.

📄 PDF Abstract BibTeX arXiv:2601.20650

Code (0)

등록된 구현이 없습니다.

SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?

2026-06-05 · Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola 외 arxiv

AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form ta…

Long-Context Understanding

LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalization

2026-06-03 · Yuanhe Zhang, Yuekai Sun, Taiji Suzuki, Jason D. Lee 외 arxiv

Long-horizon autoformalization of research mathematics fails not only at hard lemmas, but at scale: statements drift, dependencies tangle, context decays, and local repairs corrupt distant work. We present LeanMarathon, …

LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios

2026-02-03 · Tianyu Chen, Chujia Hu, Ge Gao, Dongrui Liu 외 arxiv

Computer-use agents (CUAs) that interact with real computer systems can perform automated tasks but face critical safety risks. Ambiguous instructions may trigger harmful actions, and adversarial users can manipulate too…

OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

2026-06-28 · Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu 외 arxiv

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0…

Marathon: A Race Through the Realm of Long Context with Large Language Models

2023-12-15 · Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang 외

With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models' comprehension and reasoning abilities in …

Long-Context Understanding Multiple-choice