Multi-Objective Exploration and Preference Optimization via Mutual Information
Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
This paper investigates multi-objective reinforcement learning (MORL), which focuses on learning Pareto optimal policies in the presence of multiple reward functions. Despite MORL's significant empirical success, there i…
Multi-Objective Reinforcement Learningreinforcement-learningReinforcement LearningHierarchical Soft Actor-Critic: Adversarial Exploration via Mutual Information Optimization
We describe a novel extension of soft actor-critics for hierarchical Deep Q-Networks (HDQN) architectures using mutual information metric. The proposed extension provides a suitable framework for encouraging explorations…
An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses cond…
8kMathMCCE: A Framework for Multi-LLM Collaborative Search in Discrete Spaces with Similarity-Filtered Preference Learning
Multi-objective discrete optimization problems, such as molecular design, pose significant challenges due to their vast and unstructured combinatorial spaces. Traditional evolutionary algorithms often get trapped in loca…
Reinforcement LearningApplication of Compromising Evolution in Multi-objective Image Error Concealment
Numerous multi-objective optimization problems encounter with a number of fitness functions to be simultaneously optimized of which their mutual preferences are not inherently known. Suffering from the lack of underlying…
Image Enhancement