On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment

2026-05-12 · Bo Yin, Qi Li, Xinchao Wang arxiv

Tool-using LLM agents fail through trajectories rather than only final responses, as they may execute unsafe tool calls, follow injected instructions, comply with harmful requests, or over-refuse benign tasks despite producing a seemingly safe answer. Existing safety-alignment signals are largely response-level or off-policy, and often incur a safety-utility trade-off: improving agent safety comes at the cost of degraded task performance. Such sparse and single-objective rewards severely limit real-world usability. To bridge this gap, we propose FATE, an on-policy self-evolving framework that transforms verifier-scored failures into repair supervision without expert demonstrations. For each failure, the same policy proposes repair candidates, which are then re-scored by verifiers and filtered across security, utility, over-refusal control, and trajectory validity. This dense trajectory-level information is then used as a supervision signal for agent self-evolution. During this process, we further introduce Pareto-Front Policy Optimization (PFPO), combining supervised warmup with Pareto-aware policy optimization to preserve safety-utility trade-offs. Experiments on AgentDojo, AgentHarm, and ATBench show that FATE improves safety across different models and scales while preserving useful behavior. Compared with strong baselines, FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%. These results suggest that failed trajectories can provide structured repair supervision for safer self-evolving agents.

📄 PDF Abstract BibTeX arXiv:2605.11882

Code (0)

등록된 구현이 없습니다.

FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving

2026-08-13 · Hao Dou arxiv

Reinforcement learning improves autonomous-driving vision-language-action (VLA) models by evaluating trajectories sampled from the current policy. Group relative policy optimization (GRPO) learns from reward differences …

Reinforcement Learning Autonomous Driving

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

2026-07-16 · Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang 외 arxiv

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a pract…

Reinforcement Learning Decision Making

Near-Miss: Latent Policy Failure Detection in Agentic Workflows

2026-03-31 · Ella Rabinovich, David Boaz, Naama Zwerdling, Ateret Anaby-Tavor arxiv

Agentic systems for business process automation often require compliance with policies governing conditional updates to the system state. Evaluation of policy adherence in LLM-based agentic workflows is typically perform…

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

2026-01-21 · Tianshi Xu, Yuteng Chen, Meng Li arxiv

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the explorat…

Reinforcement Learning Semantic Similarity

DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations

2026-05-23 · Lirong Che, Yuzhe yang, Peiwen lin, Chuang wang 외 arxiv

Agent harness evolution improves frozen language-model agents by modifying the executable structures around them. We study this paradigm as a form of sample-efficient fast adaptation: instead of updating model weights, a…