Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning
Recent work shows that fine-tuning language models on even a small amount of poisoned data can install targeted misbehavior, and ostensibly benign data can transmit hidden preferences that generalize broadly. Standard defenses, such as data filtering, mixing in harmless data, and regularization, attenuate these effects but do not eliminate them. We instead pursue robustness through redundancy: collecting multiple datasets from different sources and only learning what is common between them. Thus, if only a subset of sources are malicious, the misbehavior will be blocked. In order to implement this defense strategy, we fine-tune a separate reference model on each source's dataset and aggregate their next-token distributions at decoding time. We introduce two consensus decoders: a token-wise minimum, which caps each token at the lowest probability any source assigns, and a base-relative variant, which reverts to the base probability on any token the sources move in opposing directions. We further relax exact agreement to tolerate partial support across sources and different surface expressions of the same intention. Across controlled poisoning tasks, subliminal learning, and emergent misalignment, consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior, including cases where union training and weight averaging retain the unwanted behavior.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Resilient Average Consensus: A Detection and Compensation Approach
We study the problem of resilient average consensus for multi-agent systems with misbehaving nodes. To protect consensus valuefrom being influenced by misbehaving nodes, we address this problem by detecting misbehaviors,…
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
The advancement of Large Language Models (LLMs) has significantly impacted various domains, including Web search, healthcare, and software development. However, as these models scale, they become more vulnerable to cyber…
MemorizationSelf-Improving Code Generation via Semantic Entropy and Behavioral Consensus
Improving the code generation capabilities of large language models (LLMs) typically relies on supervised fine-tuning or preference optimization, both of which require costly external resources such as powerful teacher m…
Code GenerationCausal Intervention for Fairness in Multi-behavior Recommendation
Recommender systems usually learn user interests from various user behaviors, including clicks and post-click behaviors (e.g., like and favorite). However, these behaviors inevitably exhibit popularity bias, leading to s…
FairnessRecommendation SystemsSteering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing
Large language models (LLMs) are typically governed by post-training alignment (e.g., RLHF or DPO), which yields a largely static policy during deployment and inference. However, real-world safety is a full-lifecycle pro…