홈 › Papers

Direct Preference Optimization Using Sparse Feature-Level Constraints

2024-11-12 · Qingyu Yin, Chak Tou Leong, Hongbo Zhang, Minjun Zhu, Hanqi Yan, Qiang Zhang, Yulan He, Wenjie Li, Jun Wang, Yue Zhang, Linyi Yang

The alignment of large language models (LLMs) with human preferences remains a key challenge. While post-training techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have achieved notable success, they often introduce computational inefficiencies and training instability. In this paper, we propose Feature-level constrained Preference Optimization (FPO), a novel method designed to simplify the alignment process while ensuring stability. FPO leverages pre-trained Sparse Autoencoders (SAEs) and introduces feature-level constraints, allowing for efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using sparse features activated in a well-trained sparse autoencoder and the quality of sequential KL divergence by using the feature-level offline reference. Experimental results on benchmark datasets demonstrate that FPO achieves a 5.08% absolute improvement in win rate with much lower computational cost compared to state-of-the-art baselines, making it a promising solution for efficient and controllable LLM alignments.

📄 PDF Abstract BibTeX arXiv:2411.07618

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Sparse Autoencoder A Sparse Autoencoder is a type of autoencoder that employs sparsity to achieve an information bottleneck. Specifically the loss function is constructed so that activations are…

The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features

2025-09-16 · Jeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo arxiv

Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight ad…

Reinforcement Learning

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment

2026-03-07 · Jilong Liu, Yonghui Yang, Pengyang Shao, Wenjian Tao 외 arxiv

Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods of…

SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

2024-10-07 · Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou-Ammar 외

Preference Optimization (PO) has proven an effective step for aligning language models to human-desired behaviors. Current variants, following the offline Direct Preference Optimization objective, have focused on a stric…

Code Generation Text Summarization Text-to-Code Generation

Interpretable Reward Model via Sparse Autoencoder

2025-08-12 · Shuyi Zhang, Wei Shi, Sihang Li, Jiayi Liao 외 arxiv

Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with …

Reinforcement Learning

SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder

2025-11-11 · Dengcan Liu, Jiahao Li, Zheren Fu, Yi Tu 외 arxiv

Reward models (RMs) are a core component in the post-training of large language models (LLMs), serving as proxies for human preference evaluation and guiding model alignment. However, training reliable RMs under limited …