LamPO: A Lambda Style Policy Optimization for Reasoning Language Models

2026-05-20 · Redacted by arXiv arxiv

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving reasoning language models on tasks such as mathematics, coding, and scientific question answering. However, widely used group-relative objectives, such as GRPO, summarize each sampled group with scalar statistics and therefore discard fine-grained relational information among candidate responses. This weakens credit assignment under sparse outcome rewards, especially when multiple generated solutions differ only subtly in reasoning quality. We propose \textbf{LamPO}, a \textbf{Lambda-Style Policy Optimization} method that replaces scalar group advantages with a \emph{Pairwise Decomposed Advantage}. LamPO aggregates pairwise reward gaps within each response group and modulates each comparison by a confidence-aware weight computed from sequence log-probability differences, while retaining the critic-free and clipped-update structure of PPO-style optimization. When reference solutions are available, we further add a lightweight ROUGE-L-based dense auxiliary reward to reduce reward sparsity. Experiments on AIME24, AIME25, MATH-500, and GPQA-Diamond with Qwen3-1.7B, Qwen3-4B, and Phi-4-mini show that LamPO consistently improves over GRPO and recent RLVR variants, with more stable training dynamics and better sample efficiency.

📄 PDF Abstract BibTeX arXiv:2605.21235

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Question Answering

LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models

2026-05-19 · Redacted by arXiv arxiv

Group Relative Policy Optimization(GRPO) has become a cornerstone of modern reinforcement learning alignment, prized for its efficacy in foregoing an explicit value-critic by leveraging reward normalization across sample…

Reinforcement Learning

Contextual Latent-Movements Off-Policy Optimization for Robotic Manipulation Skills

2020-10-26 · Samuele Tosatto, Georgia Chalvatzaki, Jan Peters

Parameterized movement primitives have been extensively used for imitation learning of robotic tasks. However, the high-dimensionality of the parameter space hinders the improvement of such primitives in the reinforcemen…

Imitation Learning Reinforcement Learning (RL)

An Adaptive Clipping Approach for Proximal Policy Optimization

2018-04-17 · Gang Chen, Yiming Peng, Mengjie Zhang

Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies tr…

Learning Theory Reinforcement Learning

GLAMpoints: Greedily Learned Accurate Match points

2019-08-19 · ICCV 2019 10 · Prune Truong, Stefanos Apostolopoulos, Agata Mosinska, Samuel Stucky 외

We introduce a novel CNN-based feature point detector - GLAMpoints - learned in a semi-supervised manner. Our detector extracts repeatable, stable interest points with a dense coverage, specifically designed to maximize …

Image Registration Keypoint Detection

LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

2024-08-06 · Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu 외

We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples wi…

Hate Speech Detection Ordinal Classification