P^2O: Joint Policy and Prompt Optimization

2026-03-23 · Xinyu Lu, Kaiqi Zhang, Jinglin Yang, Boxi Cao, Yaojie Lu, Hongyu Lin, Min He, Xianpei Han, Le Sun arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but suffers from advantage collapse on ``hard samples'' where all rollouts fail. This lack of variance eliminates crucial learning signals. For these intractable samples, simply scaling up rollout budgets offers limited gains. We introduce Joint Policy and Prompt Optimization (P$^2$O) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. P$^2$O leverages the GEPA algorithm to discover successful reasoning prompts for intractable instances. Via context distillation, the model internalizes these prompt-induced gains directly into its parameters, removing the need for inference-time prompting. Empirically, P$^2$O restores critical advantage signals, significantly outperforming standard GRPO and surpassing baselines with doubled rollout budgets, ultimately yielding strong out-of-distribution generalization and an up to $9.5\%$ performance improvement. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates establishes a self-reinforcing paradigm for autonomous LLM alignment.

📄 PDF Abstract BibTeX arXiv:2603.21877

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Improving Retrospective Language Agents via Joint Policy Gradient Optimization

2025-03-03 · Xueyang Feng, Bo Lan, Quanyu Dai, Lei Wang 외

In recent research advancements within the community, large language models (LLMs) have sparked great interest in creating autonomous agents. However, current prompt-based agents often heavily rely on large-scale LLMs. M…

Decision Making Imitation Learning Task Planning

Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learning

2025-05-26 · Ziyi Zhang, Li Shen, Deheng Ye, Yong Luo 외

Text-to-multiview (T2MV) generation, which produces coherent multiview images from a single text prompt, remains computationally intensive, while accelerated T2MV methods using few-step diffusion models often sacrifice i…

Denoising reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

BLO-Inst: Bi-Level Optimization Based Alignment of YOLO and SAM for Robust Instance Segmentation

2026-01-29 · Li Zhang, Pengtao Xie arxiv

The Segment Anything Model has revolutionized image segmentation with its zero-shot capabilities, yet its reliance on manual prompts hinders fully automated deployment. While integrating object detectors as prompt genera…

Instance Segmentation Image Segmentation

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

2026-05-29 · Stephane Hatgis-Kessell, Emma Brunskill arxiv

We study when large language models (LLMs) can serve as effective black-box policy optimizers for reinforcement learning (RL) tasks, i.e., when can we replace classical RL algorithms with an LLM? We explore this question…

Reinforcement Learning Continuous Control

MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models

2026-04-18 · Zhaokang Liao, Yingguo Gao, Yi Yang, Yongheng Hu 외 arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising approach to improve the reasoning abilities of Large Language Models (LLMs). Among RLVR algorithms, Group Relative Policy Optimization (GRP…