Reinforcement Learning with Heterogeneous Data: Estimation and Inference
Reinforcement Learning (RL) has the promise of providing data-driven support for decision-making in a wide range of problems in healthcare, education, business, and other domains. Classical RL methods focus on the mean of the total return and, thus, may provide misleading results in the setting of the heterogeneous populations that commonly underlie large-scale datasets. We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of a given policy, and the Auto-Clustered Policy Iteration (ACPI) for estimating the optimal policy in a given policy class. Our auto-clustered algorithms can automatically detect and identify homogeneous sub-populations, while estimating the Q function and the optimal policy for each sub-population. We establish convergence rates and construct confidence intervals for the estimators obtained by the ACPE and ACPI. We present simulations to support our theoretical findings, and we conduct an empirical study on the standard MIMIC-III dataset. The latter analysis shows evidence of value heterogeneity and confirms the advantages of our new method.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Heterogeneous Agent Collaborative Reinforcement Learning
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimi…
Multi-agent Reinforcement LearningEstimating Heterogenous Treatment Effects for Survival Data with Doubly Doubly Robust Estimator
In this paper, we introduce a doubly doubly robust estimator for the average and heterogeneous treatment effect for left-truncated-right-censored (LTRC) survival data. In causal inference for survival functions in LTRC s…
Causal InferenceSurvival AnalysisUnconditional Quantile Regression with High Dimensional Data
This paper considers estimation and inference for heterogeneous counterfactual effects with high-dimensional data. We propose a novel robust score for debiased estimation of the unconditional quantile regression (Firpo, …
counterfactualquantile regressionregressionVocal Bursts Intensity PredictionUncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
We study estimation and statistical inference for reward models used in aligning large language models (LLMs). A key component of LLM alignment is reinforcement learning from human feedback (RLHF), where humans compare p…
Reinforcement LearningFrom interpretability to inference: an estimation framework for universal approximators
We present a novel framework for estimation and inference with the broad class of universal approximators. Estimation is based on the decomposition of model predictions into Shapley values. Inference relies on analyzing …