Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Large Language Models (LLMs) are increasingly deployed in applications that must weigh clashing moral values, yet even strong models exhibit hidden biases and brittle instruction-following across languages. We introduce a 12,000-instance dataset of two-option dilemmas covering pairwise three value conflicts: Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty, along with their translations into Hindi, Arabic, Spanish, and Chinese, to probe cross-lingual behavior. Benchmarking on GPT-5-mini reveals that it consistently favors Honesty over Autonomy across all five languages when no policy is given. The Llama-3.2-1/3B models exhibit strong first-option bias; however, both plain fine-tuning and Direct Preference Optimization fine-tuning effectively remove this bias, increasing accuracy to greater than 98%. In order to decouple the effect of learning correlations in the dataset from abstract values, we propose a task vector transfer based experiment where after computing the task vectors for a direction of value preference we orthogonalize it with respect to the general instruction following vector. Our experiment shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.
Code (3)
Tasks
Instruction FollowingSimilar Papers 제목 키워드 기반
The role of ethical consumption in promoting democratic sustainability: revisiting neoclassical economics through Kantian ethics
This paper explores how ethical consumption can transform democratic governance toward sustainability by challenging traditional economic models centered on utility and efficiency. As societal values shift toward transpa…
EthicsA Societal Sentiment Analysis: Predicting the Values and Ethics of Individuals by Analysing Social Media Content
To find out how users{'} social media behaviour and language are related to their ethical practices, the paper investigates applying Schwartz{'} psycholinguistic model of societal sentiment to social media text. The anal…
EthicsGeneral ClassificationregressionSentiment Analysis+1Decomposition Ascribed Synergistic Learning for Unified Image Restoration
Learning to restore multiple image degradations within a single model is quite beneficial for real-world applications. Nevertheless, existing works typically concentrate on regarding each degradation independently, while…
DeblurringDenoisingImage DeblurringImage Dehazing+6Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models
How do latent and inference time computations enable large language models (LLMs) to solve multi-step reasoning? We introduce a framework for tracing and steering algorithmic primitives that underlie model reasoning. Our…
Steering Geometry: Validating Human Value Geometry in LLM Steering Space
As large language models (LLMs) are increasingly deployed in alignment-sensitive contexts, activation steering has emerged as a lightweight, inference-time alternative to fine-tuning methods (e.g., RLHF, DPO) for behavio…