Does Representation Intervention Really Identify Desired Concepts and Elicit Alignment?
Representation intervention aims to locate and modify the representations that encode the underlying concepts in Large Language Models (LLMs) to elicit the aligned and expected behaviors. Despite the empirical success, it has never been examined whether one could locate the faithful concepts for intervention. In this work, we explore the question in safety alignment. If the interventions are faithful, the intervened LLMs should erase the harmful concepts and be robust to both in-distribution adversarial prompts and the out-of-distribution (OOD) jailbreaks. While it is feasible to erase harmful concepts without degrading the benign functionalities of LLMs in linear settings, we show that it is infeasible in the general non-linear setting. To tackle the issue, we propose Concept Concentration (COCA). Instead of identifying the faithful locations to intervene, COCA refractors the training data with an explicit reasoning process, which firstly identifies the potential unsafe concepts and then decides the responses. Essentially, COCA simplifies the decision boundary between harmful and benign representations, enabling more effective linear erasure. Extensive experiments with multiple representation intervention methods and model architectures demonstrate that COCA significantly reduces both in-distribution and OOD jailbreak success rates, and meanwhile maintaining strong performance on regular tasks such as math and code generation.
Code (0)
등록된 구현이 없습니다.
Tasks
Code GenerationMathSafety AlignmentSimilar Papers 제목 키워드 기반
On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?
Large Language Models (LLMs) have achieved unprecedented performance in Natural Language Generation (NLG) tasks. However, many existing studies have shown that they could be misused to generate undesired content. In resp…
Text GenerationMatching a Desired Causal State via Shift Interventions
Transforming a causal system from a given initial state to a desired target state is an important task permeating multiple fields including control theory, biology, and materials science. In causal models, such transform…
Active LearningSteering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models
Aligned representations across languages is a desired property in multilingual large language models (mLLMs), as alignment can improve performance in cross-lingual tasks. Typically alignment requires fine-tuning a model,…
RetrievalActive Learning for Optimal Intervention Design in Causal Models
Sequential experimental design to discover interventions that achieve a desired outcome is a key problem in various domains including science, engineering and public policy. When the space of possible interventions is la…
Active LearningExperimental DesignCost-effectively Identifying Causal Effect When Only Response Variable Observable
In many real tasks, we care about how to make decisions other than mere predictions on an event, e.g. how to increase the revenue next month instead of knowing it will drop. The key is to identify the causal effects on t…
Decision Making