From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

2026-05-11 · Pedro Conde, Henrique Branquinho, Valerio Mazzone, Bruno Mendes, André Baptista, Nuno Moniz arxiv

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.

📄 PDF Abstract BibTeX arXiv:2605.10834

Code (0)

등록된 구현이 없습니다.

Autonomous Penetration Testing using Reinforcement Learning

2019-05-15 · Jonathon Schwartz, Hanna Kurniawati

Penetration testing (pentesting) involves performing a controlled attack on a computer system in order to assess it's security. Although an effective method for testing security, pentesting requires highly skilled practi…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents

2026-07-22 · Or Zion Eliav, Eyal Lenga, Shir Bernstien, Yisroel Mirsky arxiv

Traditional pentesting uses reconnaissance at each step to uncover unseen weaknesses, build stronger attacks, and advance the objective; we argue that AI agents require the same treatment. We formalize agent reconnaissan…

Mind the Gap: Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning

2024-12-05 · Shicheng Zhou, Jingju Liu, Yuliang Lu, Jiahai Yang 외

With increasing numbers of vulnerabilities exposed on the internet, autonomous penetration testing (pentesting) has emerged as a promising research area. Reinforcement learning (RL) is a natural fit for studying this top…

Large Language Model Meta Reinforcement Learning Reinforcement Learning (RL)

Generative Artificial Intelligence-Supported Pentesting: A Comparison between Claude Opus, GPT-4, and Copilot

2025-01-12 · Antonio López Martínez, Alejandro Cano, Antonio Ruiz-Martínez

The advent of Generative Artificial Intelligence (GenAI) has brought a significant change to our society. GenAI can be applied across numerous fields, with particular relevance in cybersecurity. Among the various areas o…

Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis

2026-05-06 · Yasod Ginige, Pasindu Marasinghe, Sajal Jain, Suranga Seneviratne arxiv

Cyber threats are rapidly increasing, expanding their impact from large-scale enterprises to government services and individual users, making robust security systems increasingly essential. However, a significant shortag…

Reinforcement Learning Logical Reasoning