ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

2023-10-16 · Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, Zhi-Quan Luo

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value model as in PPO and is further enhanced with a new variance reduction technique. ReMax offers several benefits over PPO: it is simpler to implement, eliminates more than 4 hyper-parameters in PPO, reduces GPU memory usage, and shortens training time. ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO. Applying ReMax to a Mistral-7B model resulted in a 94.78% win rate on the AlpacaEval leaderboard and a 7.739 score on MT-bench, setting a new SOTA for open-source 7B models. These results show the effectiveness of ReMax while addressing the limitations of PPO in LLMs.

📄 PDF Abstract BibTeX arXiv:2310.10505

Code (3)

liziniu/ReMax 공식 구현 pytorch

niutrans/gram pytorch

openpsi-project/realhf pytorch

Tasks

General Reinforcement LearningGPUreinforcement-learning

Methods 이 논문이 사용한 방법론

ZeRO Zero Redundancy Optimizer (ZeRO) is a sharded data parallel method for distributed training. ZeRODP removes the memory state redundancies across data-parallel processes by…

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

REINFORCE REINFORCE is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to…

REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

2025-01-04 · Jian Hu

Reinforcement Learning from Human Feedback (RLHF) has emerged as a critical approach for aligning large language models with human preferences, witnessing rapid algorithmic evolution through methods such as Proximal Poli…

Computational Efficiency

Outcome-based Reinforcement Learning to Predict the Future

2025-05-23 · Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt 외

Reinforcement learning with verifiable rewards (RLVR) has boosted math and coding in large language models, yet there has been little effort to extend RLVR into messier, real-world domains like forecasting. One sticking …

Holdout Set Math reinforcement-learning Reinforcement Learning

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

2026-05-29 · Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada, Tadashi Kozuno 외 arxiv

In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greed…

Reinforcement Learning

Finite-Time Regret Analysis of Retry-Aware Bandits

2026-05-20 · Bingkui Tong, Junpei Komiyama, Soichiro Nishimori, Paavo Parmas arxiv

We study a stochastic bandit algorithm motivated by retry-aware objectives that value the best outcome among multiple attempts, such as pass@$k$ and max@$k$. Given a posterior over arm values, ReMax chooses a sampling di…