Papers Red Teaming
“Red Teaming” 태그가 달린 논문 356편 · 필터 해제
SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback
Agent skills package instructions, executable code, and task-specific resources into reusable artifacts that agents can improve using execution feedback. The same mechanism also enables attackers to evolve malicious skil…
Red TeamingBlack-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Agentic systems are rapidly moving to production, where they read untrusted inputs, call tools with real permissions, and act autonomously, expanding the security surface beyond chat-only models. Yet standard evaluations…
Red TeamingSIR: Self-improving Red-teaming for Compute Use Agents
Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate everyday digital tasks…
Red TeamingPsychJail: Exploring Psychological Jailbreaks via Multi-Turn Persuasion of LLM Policies
Large language models (LLMs) are increasingly deployed in education, healthcare, policy advising, and other interactive settings, where users engage them as sustained social interlocutors rather than one-shot query engin…
Reinforcement LearningRed TeamingFrom Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs
Large audio-language models (LALMs) have demonstrated strong capabilities in understanding diverse audio inputs. This diversity includes low-frequency signals that are inaudible to humans but can still enter the model an…
Red TeamingAgent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming
Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing s…
Reinforcement LearningRed TeamingOpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
AI agents operate in persistent environments where early state changes can influence decisions far into the future. Unlike conventional language-model interactions, agent behavior is mediated through a shared state that …
Red TeamingGPT-Red: Automated Red Teaming via Self-Play at Scale
We introduce GPT-Red, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production sy…
Red TeamingCode Monitor Red Teaming for Public-Test-Passing Code
Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM ver…
Red TeamingWhat AI Red-Team Evaluations Can and Cannot Prove
Red-team evaluations of AI models support some claims and not others, and the boundary between the two is calculable rather than merely a matter of judgment. We define the evidential ceiling of an evaluation as the large…
Red TeamingOnline Safety Monitoring for LLMs
Despite alignment training, LLMs remain prone to generating unsafe outputs at deployment time. Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical. We study a simple …
Mathematical ReasoningRed TeamingSecuring the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming
The fast growth of open-source AI infrastructure, from model serving engines and agent platforms to the Model Context Protocol (MCP) ecosystem and the language models themselves, has outpaced the security tooling availab…
Red TeamingA Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems
Large language models are no longer only text generators. They are increasingly embedded in retrieval pipelines, enterprise assistants, coding environments, robotic systems, security-operation workflows, and autonomous a…
Red TeamingA Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Large language models (LLMs) have demonstrated remarkable performance across natural language processing tasks, yet their deployment in high-stakes applications raises critical concerns regarding reliability, safety, and…
Red TeamingThe Almost Intelligent Revolution: Options for Scaling Up Deliberation and Empowering People with AI
The increasing prominence of Large Language Models (LLMs) in public discourse presents both opportunities and challenges for democratic deliberation. While red teaming strategies help mitigate specific risks, broader con…
Red TeamingToward AI-Resilient Assessment in Computer Science Courses in an AI-Native World
AI-native course assessments in senior computer science courses and related fields should grade students by \emph{AI-resilient skill}: the ability to achieve outcomes beyond a strong AI baseline. Such assessments should …
Red TeamingPosition: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety
Modern artificial intelligence (AI) systems present profound new risks to child safety. AI is increasingly being misused to create AI-generated child sexual abuse material, facilitate child sexual exploitation, and reduc…
Red TeamingCan Data Work be Reparative?
We present an ethnographic study of an alternative approach to data work, developed by a civic-tech initiative that builds datasets for training and benchmarking online safety systems. They aim to respond to online safet…
Red TeamingLearning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tan…
Reinforcement LearningRed TeamingBeyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (and Fail) Red Team Attacks
Standard AI red teaming evaluations reduce adversarial campaigns to a single binary outcome, attack success rate (ASR), not taking into account the sequential structure of how models resist or yield to attacks. We propos…
Red Teaming