Scalable Ensembling For Mitigating Reward Overoptimisation
Reinforcement Learning from Human Feedback (RLHF) has enabled significant advancements within language modeling for powerful, instruction-following models. However, the alignment of these models remains a pressing challenge as the policy tends to overfit the learned `proxy" reward model past an inflection point of utility as measured by a `gold" reward model that is more performant -- a phenomenon known as overoptimisation. Prior work has mitigated this issue by computing a pessimistic statistic over an ensemble of reward models, which is common in Offline Reinforcement Learning but incredibly costly for language models with high memory requirements, making such approaches infeasible for sufficiently large models. To this end, we propose using a shared encoder but separate linear heads. We find this leads to similar performance as the full ensemble while allowing tremendous savings in memory and time required for training for models of similar size.
Code (0)
등록된 구현이 없습니다.
Tasks
Instruction FollowingLanguage ModelingLanguage Modellingreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
SCE: Scalable Consistency Ensembles Make Blackbox Large Language Model Generation More Reliable
Large language models (LLMs) have demonstrated remarkable performance, yet their diverse strengths and weaknesses prevent any single LLM from achieving dominance across all tasks. Ensembling multiple LLMs is a promising …
Language ModelingLanguage ModellingLarge Language ModelMerging Multi-Task Models via Weight-Ensembling Mixture of Experts
Merging various task-specific Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to …
Mixture-of-ExpertsTask ArithmeticOn Cost-Effective LLM-as-a-Judge Improvement Techniques
Using a language model to score or rank candidate responses has become a scalable alternative to human evaluation in reinforcement learning from human feedback (RLHF) pipelines, benchmarking, and application layer evalua…
Reinforcement LearningPurifying Large Language Models by Ensembling a Small Language Model
The emerging success of large language models (LLMs) heavily relies on collecting abundant training data from external (untrusted) sources. Despite substantial efforts devoted to data cleaning and curation, well-construc…
Data PoisoningLanguage ModelingLanguage ModellingSmall Language ModelARES: A Reading Comprehension Ensembling Service
We introduce ARES (A Reading Comprehension Ensembling Service): a novel Machine Reading Comprehension (MRC) demonstration system which utilizes an ensemble of models to increase F1 by 2.3 points. While many of the top le…
Machine Reading ComprehensionNatural QuestionsQuestion AnsweringReading Comprehension