Offline Planning and Online Learning under Recovering Rewards

2021-06-28 · David Simchi-Levi, Zeyu Zheng, Feng Zhu

Motivated by emerging applications such as live-streaming e-commerce, promotions and recommendations, we introduce and solve a general class of non-stationary multi-armed bandit problems that have the following two features: (i) the decision maker can pull and collect rewards from up to $K\,(\ge 1)$ out of $N$ different arms in each time period; (ii) the expected reward of an arm immediately drops after it is pulled, and then non-parametrically recovers as the arm's idle time increases. With the objective of maximizing the expected cumulative reward over $T$ time periods, we design a class of ``Purely Periodic Policies'' that jointly set a period to pull each arm. For the proposed policies, we prove performance guarantees for both the offline problem and the online problems. For the offline problem when all model parameters are known, the proposed periodic policy obtains an approximation ratio that is at the order of $1-\mathcal O(1/\sqrt{K})$, which is asymptotically optimal when $K$ grows to infinity. For the online problem when the model parameters are unknown and need to be dynamically learned, we integrate the offline periodic policy with the upper confidence bound procedure to construct on online policy. The proposed online policy is proved to approximately have $\widetilde{\mathcal O}(N\sqrt{T})$ regret against the offline benchmark. Our framework and policy design may shed light on broader offline planning and online learning applications with non-stationary and recovering rewards.

📄 PDF Abstract BibTeX arXiv:2106.14813

Code (0)

등록된 구현이 없습니다.

Offline Hierarchical Reinforcement Learning via Inverse Optimization

2024-10-10 · Carolin Schmidt, Daniele Gammelli, James Harrison, Marco Pavone 외

Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. How…

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+2

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

2023-10-09 · Trevor McInroe, Adam Jelley, Stefano V. Albrecht, Amos Storkey

Offline pretraining with a static dataset followed by online fine-tuning (offline-to-online, or OtO) is a paradigm well matched to a real-world RL deployment process. In this scenario, we aim to find the best-performing …

continuous-control Continuous Control Offline RL

Learning from Visual Observation via Offline Pretrained State-to-Go Transformer

2023-06-22 · NeurIPS 2023 11 · Bohan Zhou, Ke Li, Jiechuan Jiang, Zongqing Lu

Learning from visual observation (LfVO), aiming at recovering policies from only visual observation data, is promising yet a challenging problem. Existing LfVO approaches either only adopt inefficient online learning sch…

Minecraft reinforcement-learning Reinforcement Learning

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

2023-01-05 · Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric 외

Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manuall…

continuous-control Continuous Control Self-Supervised Learning

Offline-Online Reinforcement Learning for Energy Pricing in Office Demand Response: Lowering Energy and Data Costs

2021-08-14 · Doseok Jang, Lucas Spangher, Manan Khattar, Utkarsha Agwan 외

Our team is proposing to run a full-scale energy demand response experiment in an office building. Although this is an exciting endeavor which will provide value to the community, collecting training data for the reinfor…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)