Gradient-Aware Model-based Policy Search

2019-09-09 · Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, Marcello Restelli

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introduce a novel model-based policy search approach that exploits the knowledge of the current agent policy to learn an approximate transition model, focusing on the portions of the environment that are most relevant for policy improvement. We leverage a weighting scheme, derived from the minimization of the error on the model-based policy gradient estimator, in order to define a suitable objective function that is optimized for learning the approximate transition model. Then, we integrate this procedure into a batch policy improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS), which iteratively learns a transition model and uses it, together with the collected trajectories, to compute the new policy parameters. Finally, we empirically validate GAMPS on benchmark domains analyzing and discussing its properties.

📄 PDF Abstract BibTeX arXiv:1909.04115

Code (0)

등록된 구현이 없습니다.

Tasks

modelModel-based Reinforcement LearningReinforcement Learning

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

2026-08-18 · Peizheng Guo, Jianqi Zhang, Xingyu Zhang, Yun Fan 외 arxiv

Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-bat…

On Policy Gradients

2019-11-12 · Mattis Manfred Kämmerer

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to rea…

Policy-Aware Model Learning for Policy Gradient Methods

2020-02-28 · Romina Abachi, Mohammad Ghavamzadeh, Amir-Massoud Farahmand

This paper considers the problem of learning a model in model-based reinforcement learning (MBRL). We examine how the planning module of an MBRL algorithm uses the model, and propose that the model learning module should…

model Model-based Reinforcement Learning Policy Gradient Methods Reinforcement Learning

Score-Aware Policy-Gradient Methods and Performance Guarantees using Local Lyapunov Conditions: Applications to Product-Form Stochastic Networks and Queueing Systems

2023-12-05 · Céline Comte, Matthieu Jonckheere, Jaron Sanders, Albert Senen-Cerda

In this paper, we introduce a policy-gradient method for model-based reinforcement learning (RL) that exploits a type of stationary distributions commonly obtained from Markov decision processes (MDPs) in stochastic netw…

Form Model-based Reinforcement Learning Policy Gradient Methods Reinforcement Learning (RL)

Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry

2026-05-18 · Yevhen Shcherbinin, Arina Redina, Maxim Kalpin, Vlad Kochetov arxiv

Multi-agent policy-gradient methods have been shown to converge locally near stable Nash equilibria. Local convergence, however, does not determine which equilibrium is reached. We study this question through basin-entry…