홈 › Papers

LinuxArena: A Control Setting for AI Agents in Live Production Software Environments

2026-04-16 · Tyler Tracy, Ram Potham, Nick Kuhn, Myles Heller, Anshul Khandelwal, Cody Rushing, Henri Lemoine, Miguel Brandao, Tomas Turlik, Adam Hanson, Josh Hills, Amy Ngo, Ram Rachum, Nik Mitchell, Falko Galperin, Oscar Sykes, Pip Arnott, Samuel Prieto Lima, Carlos Giudice, Matt Goldwater, Daniel Popp, Drew de Wet, Ruben Castaing, Qi Guo, Douw Marx, Benjamin Shaffrey, Justin Shenk, Martin Milbradt, Hannah Meagher, Shaheen Ahmed-Chowdhury, Daniel O'Connell, Chris Canal, Buck Shlegeris, Aryan Bhatt arxiv

We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1\% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols.

📄 PDF Abstract BibTeX arXiv:2604.15384

Code (0)

등록된 구현이 없습니다.

Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety

2026-06-03 · Catherine Ge-Wang, Tyler Crosse, Benjamin Hadad, Joachim Schaeffer 외 arxiv

An attacker that strategically chooses when to attack is much harder to catch than one that attacks indiscriminately. AI control is a safety framework for deploying capable but untrusted AI agents under the oversight of …

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection

2026-05-18 · Lei Zhao, Abhay Bhaskar, Edgar Dobriban arxiv

AI agents such as OpenClaw are increasingly deployed in local workflows with access to external tools. This creates indirect prompt-injection (IPI) risk: an agent may execute harmful instructions embedded in untrusted in…

Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

2026-06-22 · Julia Belikova, Rauf Parchiev, Evgeny Egorov, Grigorii Davydenko 외 hf

Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise…

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

2025-07-10 · Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei 외 arxiv

Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously. To bridge this gap, we introduce StarDojo, a…

Multimodal Reasoning

Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

2026-06-07 · Aman Gupta, Kevin Rossell, Edesio Alcobaça, Jose Chrystian Lima Pacheco 외 arxiv

The rapid rise in LLM capabilities has made AI agents increasingly viable across a broad range of tasks. Among the most promising applications is building production-ready customer-facing agents, a challenge that demands…