Improving LLM Safety Alignment with Dual-Objective Optimization

2025-03-05 · Xuandong Zhao, Will Cai, Tianneng Shi, David Huang, Licong Lin, Song Mei, Dawn Song

Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed alignment method, exhibits limitations in both experimental and theoretical contexts as its loss function proves suboptimal for refusal learning. Through gradient-based analysis, we identify these shortcomings and propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge. This approach significantly increases LLM robustness against a wide range of jailbreak attacks, including prefilling, suffix, and multi-turn attacks across both in-distribution and out-of-distribution scenarios. Furthermore, we introduce a method to emphasize critical refusal tokens by incorporating a reward-based token-level weighting mechanism for refusal learning, which further improves the robustness against adversarial exploits. Our research also suggests that robustness to jailbreak attacks is correlated with token distribution shifts in the training process and internal representations of refusal and harmful tokens, offering valuable directions for future research in LLM safety alignment. The code is available at https://github.com/wicai24/DOOR-Alignment

📄 PDF Abstract BibTeX arXiv:2503.03710

Code (1)

wicai24/door-alignment 공식 구현 pytorch

Tasks

Safety Alignment

Methods 이 논문이 사용한 방법론

DPO 설명 없음

Multi-Objective Alignment of Language Models for Personalized Psychotherapy

2026-02-17 · Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna, Sriram Sankararaman 외 arxiv

Mental health disorders affect over 1 billion people worldwide, yet access to care remains limited by workforce shortages and cost constraints. While AI systems show therapeutic promise, current alignment approaches opti…

Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control

2026-02-07 · Yonghui Yang, Wenjian Tao, Jilong Liu, Xingyu Zhu 외 arxiv

Safety alignment of large language models remains brittle under domain shift and noisy preference supervision. Most existing robust alignment methods focus on uncertainty in alignment data, while overlooking optimization…

Mitigating Safety Fallback in Editing-based Backdoor Injection on LLMs

2025-06-16 · Houcheng Jiang, Zetong Zhao, Junfeng Fang, Haokai Ma 외

Large language models (LLMs) have shown strong performance across natural language tasks, but remain vulnerable to backdoor attacks. Recent model editing-based approaches enable efficient backdoor injection by directly m…

Diversity Model Editing Safety Alignment

Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

2024-10-25 · Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou 외

Balancing helpfulness and safety (harmlessness) is a critical challenge in aligning large language models (LLMs). Current approaches often decouple these two objectives, training separate preference models for helpfulnes…