Smooth Gate Functions for Soft Advantage Policy Optimization

2026-02-22 · Egor Denisov, Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko arxiv

Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

📄 PDF Abstract BibTeX arXiv:2602.19345

Code (0)

등록된 구현이 없습니다.

Tasks

Mathematical Reasoning

Soft Deterministic Policy Gradient with Gaussian Smoothing

2026-05-07 · Hyunjun Na, Donghwan Lee arxiv

Deterministic policy gradient (DPG) is widely utilized for continuous control; however, it inherently relies on the differentiability of the critic with respect to the action during policy updates. This assumption is vio…

Reinforcement Learning Continuous Control

Reward-Punishment Reinforcement Learning with Maximum Entropy

2024-05-20 · Jiexin Wang, Eiji Uchibe

We introduce the ``soft Deep MaxPain'' (softDMP) algorithm, which integrates the optimization of long-term policy entropy into reward-punishment reinforcement learning objectives. Our motivation is to facilitate a smooth…

reinforcement-learning Reinforcement Learning

Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)

2020-02-07 · Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li 외

The optimal policy of a reinforcement learning problem is often discontinuous and non-smooth. I.e., for two states with similar representations, their optimal policies can be significantly different. In this case, repres…

continuous-control Continuous Control Mixture-of-Experts Reinforcement Learning

Optimal Approximation - Smoothness Tradeoffs for Soft-Max Functions

2020-12-01 · NeurIPS 2020 12 · Alessandro Epasto, Mohammad Mahdian, Vahab Mirrokni, Emmanouil Zampetakis

A soft-max function has two main efficiency measures: (1) approximation - which corresponds to how well it approximates the maximum function, (2) smoothness - which shows how sensitive it is to changes of its input. Our …

Optimal Approximation -- Smoothness Tradeoffs for Soft-Max Functions

2020-10-22 · Alessandro Epasto, Mohammad Mahdian, Vahab Mirrokni, Manolis Zampetakis