Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes

2025-01-06 · Zijian Wang, Bin Wang, Mingwen Shao, Hongbo Dou, Boxiang Tao

Hybrid action models are widely considered an effective approach to reinforcement learning (RL) modeling. The current mainstream method is to train agents under Parameterized Action Markov Decision Processes (PAMDPs), which performs well in specific environments. Unfortunately, these models either exhibit drastic low learning efficiency in complex PAMDPs or lose crucial information in the conversion between raw space and latent space. To enhance the learning efficiency and asymptotic performance of the agent, we propose a model-based RL (MBRL) algorithm, FLEXplore. FLEXplore learns a parameterized-action-conditioned dynamics model and employs a modified Model Predictive Path Integral control. Unlike conventional MBRL algorithms, we carefully design the dynamics loss function and reward smoothing process to learn a loose yet flexible model. Additionally, we use the variational lower bound to maximize the mutual information between the state and the hybrid action, enhancing the exploration effectiveness of the agent. We theoretically demonstrate that FLEXplore can reduce the regret of the rollout trajectory through the Wasserstein Metric under given Lipschitz conditions. Our empirical results on several standard benchmarks show that FLEXplore has outstanding learning efficiency and asymptotic performance compared to other baselines.

📄 PDF Abstract BibTeX arXiv:2501.02774

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning (RL)

Parameterized Exploration

2019-07-13 · Jesse Clifton, Lili Wu, Eric Laber

We introduce Parameterized Exploration (PE), a simple family of methods for model-based tuning of the exploration schedule in sequential decision problems. Unlike common heuristics for exploration, our method accounts fo…

Multi-Armed Bandits

Worst-Case Regret Bounds for Exploration via Randomized Value Functions

2019-06-07 · NeurIPS 2019 12 · Daniel Russo

This paper studies a recent proposal to use randomized value functions to drive exploration in reinforcement learning. These randomized value functions are generated by injecting random noise into the training data, maki…

Efficient Exploration reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Active exploration in parameterized reinforcement learning

2016-10-06 · Mehdi Khamassi, Costas Tzafestas

Online model-free reinforcement learning (RL) methods with continuous actions are playing a prominent role when dealing with real-world applications such as Robotics. However, when confronted to non-stationary environmen…

Meta-Learning Q-Learning reinforcement-learning Reinforcement Learning+1

From Relative Entropy to Minimax: A Unified Framework for Coverage in MDPs

2026-01-17 · Xihe Gu, Urbashi Mitra, Tara Javidi arxiv

Targeted and deliberate exploration of state--action pairs is essential in reward-free Markov Decision Problems (MDPs). More precisely, different state-action pairs exhibit different degree of importance or difficulty wh…

Meta-Learning Parameterized Skills

2022-06-07 · Haotian Fu, Shangqun Yu, Saket Tiwari, Michael Littman 외

We propose a novel parameterized skill-learning algorithm that aims to learn transferable parameterized skills and synthesize them into a new action space that supports efficient learning in long-horizon tasks. We propos…

Meta-Learning Robot Manipulation