The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs

2025-11-25 · Craig Dickson arxiv

Prior work has shown that fine-tuning models on a narrow domain with misaligned data can lead to broad misalignment - a phenomenon termed "emergent misalignment" (Betley et al. 2025). While all tested models were susceptible to emergent misalignment, some models showed more resistance than others. Specifically the Qwen-2.5 family proved to be relatively resistant, while GPT-4o exhibited the strongest misalignment. In this paper we evaluate if current-generation open-weights models exhibit similar resistance to the Qwen-2.5 family and measure misalignment robustness over a range of model architectures and scales. We replicate the effect across nine modern open-weights models (Gemma 3 and Qwen 3 families, 1B-32B parameters). Models fine-tuned on insecure code generation show a 0.68% misalignment rate (compared to 0.07% for base models), matching the lower end of prior open-model results but dramatically lower than GPT-4o's 20%. We identify a critical format-dependent vulnerability: requiring JSON output doubles misalignment rates compared to natural language prompts (0.96% vs 0.42%). This suggests that structural constraints may bypass safety training by reducing the model's 'degrees of freedom' to refuse. These findings confirm emergent misalignment as a reproducible phenomenon in modern open-weights models, with rates substantially lower than observed in proprietary systems.

📄 PDF Abstract BibTeX arXiv:2511.20104

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

2024-01-16 · Xinni Jiang, Zengsheng Kuang, Chunle Guo, Ruixun Zhang 외

Guided depth super-resolution (GDSR) involves restoring missing depth details using the high-resolution RGB image of the same scene. Previous approaches have struggled with the heterogeneity and complementarity of the mu…

cross-modal alignment feature selection Super-Resolution

Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation

2026-06-07 · Qi Cao, Jian Lou, Meiting Liu, Wenjie Feng 외 arxiv

Activation steering has emerged as a popular inference-time technique for modulating the behavior of large language models (LLMs). By constructing a steering vector from examples of a target behavior and injecting it int…

Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence

2026-04-09 · Niklas Herbster, Martin Zborowski, Alberto Tosato, Gauthier Gidel 외 arxiv

Alignment in LLMs is more brittle than commonly assumed: misalignment can be induced by adversarial prompts, benign fine-tuning, emergent misalignment, and goal misgeneralization. Recent evidence suggests that some misal…

BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking

2026-01-31 · Muhammed Ustaomeroglu, Guannan Qu arxiv

Emergent misalignment can arise when a language model is fine-tuned on a narrowly scoped supervised objective: the model learns the target behavior, yet also develops undesirable out-of-domain behaviors. We investigate a…

Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures

2026-01-30 · Yanghao Su, Wenbo Zhou, Tianwei Zhang, Qiu Han 외 arxiv

Emergent Misalignment refers to a failure mode in which fine-tuning large language models (LLMs) on narrowly scoped data induces broadly misaligned behavior. Prior explanations mainly attribute this phenomenon to the gen…