Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning

2026-04-02 · Yuhui Chen, Haoran Li, Zhennan Jiang, Yuxing Qin, Yuxuan Wan, Weiheng Liu, Dongbin Zhao arxiv

Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/.

📄 PDF Abstract BibTeX arXiv:2604.01860

Code (0)

등록된 구현이 없습니다.

Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)

2023-02-01 · Amartya Mukherjee, Jun Liu

This paper introduces the Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO) algorithm into reinforcement learning. The Hamilton-Jacobi-Bellman (HJB) equation is used in control theory to evaluate the optimali…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Near-Optimal Streaming Heavy-Tailed Statistical Estimation with Clipped SGD

2024-10-26 · Aniket Das, Dheeraj Nagaraj, Soumyabrata Pal, Arun Suggala 외

We consider the problem of high-dimensional heavy-tailed statistical estimation in the streaming setting, which is much harder than the traditional batch setting due to memory constraints. We cast this problem as stochas…

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization

2023-11-10 · Jared Markowitz, Edward W. Staley

To facilitate efficient learning, policy gradient approaches to deep reinforcement learning (RL) are typically paired with variance reduction measures and strategies for making large but safe policy changes based on a ba…

Deep Reinforcement Learning Multi-Task Learning Policy Gradient Methods Reinforcement Learning (RL)

The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure

2022-05-20 · Xing Chen, Dongcui Diao, Hechang Chen, Hengshuai Yao 외

The popular Proximal Policy Optimization (PPO) algorithm approximates the solution in a clipped policy space. Does there exist better policies outside of this space? By using a novel surrogate objective that employs the …

Efficient Exploration Policy Gradient Methods

Byzantine-Robust Decentralized Learning via ClippedGossip

2022-02-03 · Lie He, Sai Praneeth Karimireddy, Martin Jaggi

In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs. Unlike federated learning where workers communicate through a server, workers in the decentralize…

Federated Learning