Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning
Dynamic mechanism design has garnered significant attention from both computer scientists and economists in recent years. By allowing agents to interact with the seller over multiple rounds, where agents' reward functions may change with time and are state-dependent, the framework is able to model a rich class of real-world problems. In these works, the interaction between agents and sellers is often assumed to follow a Markov Decision Process (MDP). We focus on the setting where the reward and transition functions of such an MDP are not known a priori, and we are attempting to recover the optimal mechanism using an a priori collected data set. In the setting where the function approximation is employed to handle large state spaces, with only mild assumptions on the expressiveness of the function class, we are able to design a dynamic mechanism using offline reinforcement learning algorithms. Moreover, learned mechanisms approximately have three key desiderata: efficiency, individual rationality, and truthfulness. Our algorithm is based on the pessimism principle and only requires a mild assumption on the coverage of the offline data set. To the best of our knowledge, our work provides the first offline RL algorithm for dynamic mechanism design without assuming uniform coverage.
Code (0)
등록된 구현이 없습니다.
Tasks
Offline RLreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach
In this article, we propose a novel pessimism-based Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the …
Thompson SamplingVariational InferenceModel-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and a dynamics model. While the dynamics model learned through reuse of the static…
D4RLOffline RLreinforcement-learningReinforcement Learning+1State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
Pessimism is of great importance in offline reinforcement learning (RL). One broad category of offline RL algorithms fulfills pessimism by explicit or implicit behavior regularization. However, most of them only consider…
Offline RLQ-Learningreinforcement-learningReinforcement Learning+1Pessimism-Free Offline Learning in General-Sum Games via KL Regularization
Offline multi-agent reinforcement learning in general-sum settings is challenged by the distribution shift between logged datasets and target equilibrium policies. While standard methods rely on manual pessimistic penalt…
Multi-agent Reinforcement LearningBeyond Pessimism: Offline Learning in KL-regularized Games
We study offline learning in KL-regularized two-player zero-sum games, where policies are optimized with respect to a fixed reference policy through KL regularization. Prior work relies on pessimistic value estimation to…