PPO in the Fisher-Rao geometry
Proximal Policy Optimization (PPO) has become a widely adopted algorithm for reinforcement learning, offering a practical policy gradient method with strong empirical performance. Despite its popularity, PPO lacks formal theoretical guarantees for policy improvement and convergence. PPO is motivated by Trust Region Policy Optimization (TRPO) that utilizes a surrogate loss with a KL divergence penalty, which arises from linearizing the value function within a flat geometric space. In this paper, we derive a tighter surrogate in the Fisher-Rao (FR) geometry, yielding a novel variant, Fisher-Rao PPO (FR-PPO). Our proposed scheme provides strong theoretical guarantees, including monotonic policy improvement. Furthermore, in the tabular setting, we demonstrate that FR-PPO achieves sub-linear convergence without any dependence on the dimensionality of the action or state spaces, marking a significant step toward establishing formal convergence results for PPO-based algorithms.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Fisher-Rao geometry of CES distributions
When dealing with a parametric statistical model, a Riemannian manifold can naturally appear by endowing the parameter space with the Fisher information metric. The geometry induced on the parameters by this metric is th…
Riemannian optimizationThe Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Models
In this work, we study the problem of choosing the discretisation schedule for sampling from masked discrete diffusion models in terms of the information geometry of the induced probability path. Specifically, we show th…
AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints
Merging large language models (LLMs) is a practical way to compose capabilities from multiple fine-tuned checkpoints without retraining. Yet standard schemes (linear weight soups, task vectors, and Fisher-weighted averag…
Supervised Quadratic Feature Analysis: Information Geometry Approach for Dimensionality Reduction
Supervised dimensionality reduction aims to map labeled data to a low-dimensional feature space while maximizing class discriminability. Directly computing discriminability is often impractical, so an alternative approac…
Dimensionality ReductionMetric LearningSupervised dimensionality reductionThe Geometry of Updates: Fisher Alignment at Vocabulary Scale
Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in predict…