ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

2026-06-23 · Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier, See-Kiong Ng, Bryan Kian Hsiang Low arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.

📄 PDF Abstract BibTeX arXiv:2606.24994

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Mathematical Reasoning

HAEPO: History-Aggregated Exploratory Policy Optimization

2025-08-26 · Gaurish Trivedi, Alakh Sharma, Kartikey Singh Bhandari, Dhruv Kumar 외 arxiv

Exploration is essential in modern learning, from reinforcement learning environments with small neural policies to large language models (LLMs). Existing work, such as DPO, leverages full sequence log-likelihoods to cap…

Reinforcement Learning

CoLVR: Enhancing Exploratory Latent Visual Reasoning via Contrastive Optimization

2026-05-09 · Ziyang Ding, Linjian Meng, Yiming Wu, Yuhan Li 외 arxiv

Due to the potential for exploratory reasoning of Latent Visual Reasoning, recent works tend to enable MLLMs (Multimodal Large Language Models) to perform visual reasoning by propagating continuous hidden states instead …

Reinforcement Learning Visual Reasoning

Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization

2025-06-03 · Marco Calì, Giulio Giacomuzzo, Ruggero Carli, Alberto Dalla Libera

This paper addresses the slow policy optimization convergence of Monte Carlo Probabilistic Inference for Learning Control (MC-PILCO), a state-of-the-art model-based reinforcement learning (MBRL) algorithm, by integrating…

Model-based Reinforcement Learning

Learning Structured Reasoning via Tractable Trajectory Control

2026-03-02 · Po-Nien Kung, Zhen Yang, Jeffrey Luo, Cheng-Fu Yang 외 arxiv

Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning trajectories remain sparse in unconstrain…

Mathematical Reasoning

S-GRPO: Unified Post-Training for Large Vision-Language Models

2026-04-17 · Yuming Yan, Kai Tang, Sihong Chen, Ke Xu 외 arxiv

Current post-training methodologies for adapting Large Vision-Language Models (LVLMs) generally fall into two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Despite their prevalence, both approa…

Reinforcement Learning Domain Adaptation