MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora, making them powerful tools for various applications. To make LLMs more usable, aligning them with human preferences is essential. Existing alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), typically embed predefined preferences directly within the model's parameters. These methods, however, often result in a static alignment that can not account for the diversity of human preferences in practical applications. In response to this challenge, we propose an effective method, \textbf{MetaAlign}, which aims to help LLMs dynamically align with various explicit or implicit preferences specified at inference time. Experimental results show that LLMs optimized on our meticulously constructed MetaAlign Dataset can effectively align with any preferences specified at the inference stage, validating the feasibility of MetaAlign. We hope that our work can provide some insights into the alignment of language models.
Code (1)
Tasks
DiversityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model p…
GPUIn-Context LearningMetaAlign: Coordinating Domain Alignment and Classification for Unsupervised Domain Adaptation
For unsupervised domain adaptation (UDA), to alleviate the effect of domain shift, many approaches align the source and target domains in the feature space by adversarial learning or by explicitly aligning their statisti…
ClassificationDomain AdaptationGeneral ClassificationMeta-Learning+4On Diversified Preferences of Large Language Model Alignment
Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotato…
Language ModelingLanguage ModellingLarge Language ModelBAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous know…
Continual LearningGeneral KnowledgeKnowledge DistillationPersonalized and Emotional Conversation+1ABC Align: Large Language Model Alignment for Safety & Accuracy
Alignment of Large Language Models (LLMs) remains an unsolved problem. Human preferences are highly distributed and can be captured at multiple levels of abstraction, from the individual to diverse populations. Organisat…
Language ModelingLanguage ModellingLarge Language ModelSynthetic Data Generation