ALaRM: Align Language Models via Hierarchical Rewards Modeling
We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.
Code (1)
Tasks
Long Form Question AnsweringMachine TranslationQuestion AnsweringText GenerationSimilar Papers 제목 키워드 기반
Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behav…
Reinforcement LearningHierarchical Latent Reasoning for LLM-based Recommendation
Large Language Models (LLMs) have shown strong potential for recommendation by leveraging their semantic understanding and contextual modeling capabilities. Recent studies further introduce reasoning mechanisms to improv…
Representation LearningVision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learning
Guiding the policy of multi-agent reinforcement learning to align with human common sense is a difficult problem, largely due to the complexity of modeling common sense as a reward, especially in complex and long-horizon…
Common Sense ReasoningLanguage ModelingLanguage ModellingLarge Language Model+1Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models
Reinforcement Learning from Verifiable Rewards (RLVR) has substantially enhanced the reasoning capabilities of large language models in abstract reasoning tasks. However, its application to Large Vision-Language Models (…
Reinforcement LearningMultimodal ReasoningFrom Evaluation to Defense: Advancing Safety in Video Large Language Models
While the safety risks of image-based large language models have been extensively studied, their video-based counterparts (Video LLMs) remain critically under-examined. To systematically study this problem, we introduce …
Safety Alignment