Value Drifts: Tracing Value Alignment During LLM Post-Training
As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.
Code (0)
등록된 구현이 없습니다.
Tasks
General KnowledgeSimilar Papers 제목 키워드 기반
VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
Aligning Large Language Models (LLMs) with nuanced human values remains a critical challenge, as existing methods like Reinforcement Learning from Human Feedback (RLHF) often handle only coarse-grained attributes. In pra…
Reinforcement LearningTracing and Manipulating Intermediate Values in Neural Math Problem Solvers
How language models process complex input that requires multiple steps of inference is not well understood. Previous research has shown that information about intermediate values of these inputs can be extracted from the…
MathSimplified stochastic calculus with applications in Economics and Finance
The paper introduces a simple way of recording and manipulating general stochastic processes without explicit reference to a probability measure. In the new calculus, operations traditionally presented in a measure-speci…
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) wit…
Comparison of neural networks and Kalman filters performances for fouling detection in a heat exchanger
This paper presents the comparison between a neural network model and a Kalman filter model when applied for fouling detection. The models are determined using data that do not require the heat exchanger to be in a stead…