Soft Sequence Policy Optimization

2026-02-22 · Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko arxiv

A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.

📄 PDF Abstract BibTeX arXiv:2602.19327

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Mathematical Reasoning

Soft Adaptive Policy Optimization

2025-11-25 · Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang 외 arxiv

Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level imp…

Reinforcement Learning Mathematical Reasoning

VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training

2026-02-11 · Guobin Shen, Chenxiao Zhao, Xiang Cheng, Lei Huang 외 arxiv

Off-policy updates are inevitable in reinforcement learning (RL) for large language models (LLMs) due to rollout staleness from asynchronous training and mismatches between training and inference engines. Naive importanc…

Reinforcement Learning Code Generation

SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization

2025-11-09 · Zhi Zheng, Yu Gu, Wei Liu, Yee Whye Teh 외 arxiv

The soft-thinking paradigm for Large Language Model (LLM) reasoning can outperform the conventional discrete-token Chain-of-Thought (CoT) reasoning in some scenarios, underscoring its research and application value. Howe…

Reinforcement Learning

Learning to Superoptimize Real-world Programs

2021-09-28 · Alex Shypula, Pengcheng Yin, Jeremy Lacomis, Claire Le Goues 외

Program optimization is the process of modifying software to execute more efficiently. Superoptimizers attempt to find the optimal program by employing significantly more expensive search and constraint solving technique…

Compiler Optimization Imitation Learning

Modified Actor-Critics

2019-07-02 · Erinc Merdivan, Sten Hanke, Matthieu Geist

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)