Multi-Objective Policy Gradients with Topological Constraints

2022-09-15 · Kyle Hollins Wray, Stas Tiomkin, Mykel J. Kochenderfer, Pieter Abbeel

Multi-objective optimization models that encode ordered sequential constraints provide a solution to model various challenging problems including encoding preferences, modeling a curriculum, and enforcing measures of safety. A recently developed theory of topological Markov decision processes (TMDPs) captures this range of problems for the case of discrete states and actions. In this work, we extend TMDPs towards continuous spaces and unknown transition dynamics by formulating, proving, and implementing the policy gradient theorem for TMDPs. This theoretical result enables the creation of TMDP learning algorithms that use function approximators, and can generalize existing deep reinforcement learning (DRL) approaches. Specifically, we present a new algorithm for a policy gradient in TMDPs by a simple extension of the proximal policy optimization (PPO) algorithm. We demonstrate this on a real-world multiple-objective navigation problem with an arbitrary ordering of objectives both in simulation and on a real robot.

📄 PDF Abstract BibTeX arXiv:2209.07096

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learning

Policy Gradients for Probabilistic Constrained Reinforcement Learning

2022-10-02 · Weiqin Chen, Dharmashankar Subramanian, Santiago Paternain

This paper considers the problem of learning safe policies in the context of reinforcement learning (RL). In particular, we consider the notion of probabilistic safety. This is, we aim to design policies that maintain th…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning

2022-06-10 · Ruida Zhou, Tao Liu, Dileep Kalathil, P. R. Kumar 외

We study policy optimization for Markov decision processes (MDPs) with multiple reward value functions, which are to be jointly optimized according to given criteria such as proportional fairness (smooth concave scalariz…

Fairness Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning

2024-05-26 · Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang 외

In numerous reinforcement learning (RL) problems involving safety-critical systems, a key challenge lies in balancing multiple objectives while simultaneously meeting all stringent safety constraints. To tackle this issu…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning

2024-03-01 · Dohyeong Kim, Mineui Hong, Jeongho Park, Songhwai Oh

In many real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines. To address these considerations, we propose a constrained multi-objective RL algor…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-Objective Reinforcement Learning for Power Grid Topology Control

2025-01-27 · Thomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn 외

Transmission grid congestion increases as the electrification of various sectors requires transmitting more power. Topology control, through substation reconfiguration, can reduce congestion but its potential remains und…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning