Offline Hierarchical Reinforcement Learning via Inverse Optimization

2024-10-10 · Carolin Schmidt, Daniele Gammelli, James Harrison, Marco Pavone, Filipe Rodrigues

Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. However, learning hierarchical policies from static offline datasets presents a significant challenge. Crucially, actions taken by higher-level policies may not be directly observable within hierarchical controllers, and the offline dataset might have been generated using a different policy structure, hindering the use of standard offline learning algorithms. In this work, we propose OHIO: a framework for offline reinforcement learning (RL) of hierarchical policies. Our framework leverages knowledge of the policy structure to solve the \textit{inverse problem}, recovering the unobservable high-level actions that likely generated the observed data under our hierarchical policy. This approach constructs a dataset suitable for off-the-shelf offline training. We demonstrate our framework on robotic and network optimization problems and show that it substantially outperforms end-to-end RL methods and improves robustness. We investigate a variety of instantiations of our framework, both in direct deployment of policies trained offline and when online fine-tuning is performed. Code and data are available at https://ohio-offline-hierarchical-rl.github.io

📄 PDF Abstract BibTeX arXiv:2410.07933

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL

2023-09-13 · Hao Sun, Alihan Hüyük, Mihaela van der Schaar

In this study, we aim to enhance the arithmetic reasoning ability of Large Language Models (LLMs) through zero-shot prompt optimization. We identify a previously overlooked objective of query dependency in such optimizat…

Arithmetic Reasoning Navigate Prompt Engineering reinforcement-learning

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning

2023-02-15 · NeurIPS 2023 11 · Siliang Zeng, Chenliang Li, Alfredo Garcia, Mingyi Hong

Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurat…

Autonomous Driving continuous-control Continuous Control D4RL+3

Offline Reinforcement Learning via Inverse Optimization

2025-02-27 · Ioannis Dimanidis, Tolga Ok, Peyman Mohajerin Esfahani

Inspired by the recent successes of Inverse Optimization (IO) across various application domains, we propose a novel offline Reinforcement Learning (ORL) algorithm for continuous state and action spaces, leveraging the c…

Model Predictive Control MuJoCo reinforcement-learning Reinforcement Learning

OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning

2025-10-17 · Woo-Jin Ahn, Sang-Ryul Baek, Yong-Jun Lee, Hyun-Duck Choi 외 arxiv

Reinforcement learning algorithms typically utilize an interactive simulator (i.e., environment) with a predefined reward function for policy training. Developing such simulators and manually defining reward functions, h…

Reinforcement Learning

Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensors

2025-09-22 · Chang Liu, Ladda Thiamwong, Yanjie Fu, Rui Xie arxiv

Utilizing offline reinforcement learning (RL) with real-world clinical data is getting increasing attention in AI for healthcare. However, implementation poses significant challenges. Defining direct rewards is difficult…

Reinforcement Learning Offline RL