ERPPO: Entropy Regularization-based Proximal Policy Optimization

2026-05-13 · Changha Lee, Gyusang Cho arxiv

Multi-Agent Proximal Policy Optimization (MAPPO) is a variant of the Proximal Policy Optimization (PPO) algorithm, specifically tailored for multi-agent reinforcement learning (MARL). MAPPO optimizes cooperative multi-agent settings by employing a centralized critic with decentralized actors. However, in case of multi-dimensional environment, MAPPO can not extract optimal policy due to non-stationary agent observation. To overcome this problem, we introduce a novel approach, Entropy Regularization-based Proximal Policy Optimization (ERPPO). For the policy optimization, we first define the object detection ambiguity under multi-dimensional observation environment. Distributional Spatiotemporal Ambiguity (DSA) learner is trained to estimate object detection uncertainty in non-stationary constraints. Then, we enhance PPO with a novel Entropy Regularization term. This regularization dynamically adjusts the policy update by applying a stronger (L1) regularization in high-ambiguity observation to encourage significant exploratory actions and a weaker (L2) regularization in low-ambiguity observation to stabilize the proximal policy optimization. This approach is designed to enhance the probability of successful object localization in time-critical operations by reducing detection failures and optimizing search policy. Experiments on a testbed with AirSim-based maritime searching scenarios show that the proposed ERPPO improves accuracy performance. Our proposed method improves higher gradient than MAPPO. Qualitative results confirm that ERPPO effectiveness in terms of suppressing false detection in visually uncertain conditions.

📄 PDF Abstract BibTeX arXiv:2605.13131

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning Object Localization Object Detection

Complexity-Regularized Proximal Policy Optimization

2025-09-24 · Luca Serfilippi, Giorgio Franceschelli, Antonio Corradi, Mirco Musolesi arxiv

Policy gradient methods usually rely on entropy regularization to prevent premature convergence. However, maximizing entropy indiscriminately pushes the policy towards a uniform distribution, often overriding the reward …

HyperPPO: A scalable method for finding small policies for robotic control

2023-09-28 · Shashank Hegde, Zhehui Huang, Gaurav S. Sukhatme

Models with fewer parameters are necessary for the neural control of memory-limited, performant robots. Finding these smaller neural network architectures can be time-consuming. We propose HyperPPO, an on-policy reinforc…

NeuFACO: Neural Focused Ant Colony Optimization for Traveling Salesman Problem

2025-09-21 · Dat Thanh Tran, Khai Quang Tran, Khoi Anh Pham, Van Khu Vu 외 arxiv

This study presents Neural Focused Ant Colony Optimization (NeuFACO), a non-autoregressive framework for the Traveling Salesman Problem (TSP) that combines advanced reinforcement learning with enhanced Ant Colony Optimiz…

Reinforcement Learning Graph Neural Network

Policy Optimization in Zero-Sum Markov Games: Fictitious Self-Play Provably Attains Nash Equilibria

2021-01-01 · Boyi Liu, Zhuoran Yang, Zhaoran Wang

Fictitious Self-Play (FSP) has achieved significant empirical success in solving extensive-form games. However, from a theoretical perspective, it remains unknown whether FSP is guaranteed to converge to Nash equilibria…

A Theory of Regularized Markov Decision Processes

2019-01-31 · Matthieu Geist, Bruno Scherrer, Olivier Pietquin

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes …

Deep Reinforcement Learning Q-Learning Reinforcement Learning