From Function to Distribution Modeling: A PAC-Generative Approach to Offline Optimization

2024-01-04 · Qiang Zhang, Ruida Zhou, Yang shen, Tie Liu

This paper considers the problem of offline optimization, where the objective function is unknown except for a collection of `offline" data examples. While recent years have seen a flurry of work on applying various machine learning techniques to the offline optimization problem, the majority of these work focused on learning a surrogate of the unknown objective function and then applying existing optimization algorithms. While the idea of modeling the unknown objective function is intuitive and appealing, from the learning point of view it also makes it very difficult to tune the objective of the learner according to the objective of optimization. Instead of learning and then optimizing the unknown objective function, in this paper we take on a less intuitive but more direct view that optimization can be thought of as a process of sampling from a generative model. To learn an effective generative model from the offline data examples, we consider the standard technique of `re-weighting", and our main technical contribution is a probably approximately correct (PAC) lower bound on the natural optimization objective, which allows us to jointly learn a weight function and a score-based generative model. The robustly competitive performance of the proposed approach is demonstrated via empirical studies using the standard offline optimization benchmarks.

📄 PDF Abstract BibTeX arXiv:2401.02019

Code (0)

등록된 구현이 없습니다.

Boosting Offline Reinforcement Learning with Residual Generative Modeling

2021-06-19 · Hua Wei, Deheng Ye, Zhao Liu, Hao Wu 외

Offline reinforcement learning (RL) tries to learn the near-optimal policy with recorded offline experience without online exploration. Current offline RL research includes: 1) generative modeling, i.e., approximating a …

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Offline Model-Based Optimization: Comprehensive Review

2025-03-21 · Minsu Kim, Jiayao Gu, Ye Yuan, Taeyoung Yun 외

Offline optimization is a fundamental challenge in science and engineering, where the goal is to optimize black-box functions using only offline datasets. This setting is particularly relevant when querying the objective…

model Neural Architecture Search scientific discovery

Scaling Offline RL via Efficient and Expressive Shortcut Models

2025-05-28 · Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen, Bradley Guo 외

Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Generative OOD-regularized Model-based Policy Optimization

2026-05-23 · Aysin Tumay, Jiahe Huang, Elise Jortberg, Rose Yu arxiv

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. T…

Reinforcement Learning Density Estimation Offline RL

Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models

2024-05-30 · Masatoshi Uehara, Yulai Zhao, Ehsan Hajiramezanali, Gabriele Scalia 외

AI-driven design problems, such as DNA/protein sequence design, are commonly tackled from two angles: generative modeling, which efficiently captures the feasible design space (e.g., natural images or biological sequence…