Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It
There has been a growing interest in off-policy evaluation in the literature such as recommender systems and personalized medicine. We have so far seen significant progress in developing estimators aimed at accurately estimating the effectiveness of counterfactual policies based on biased logged data. However, there are many cases where those estimators are used not only to evaluate the value of decision making policies but also to search for the best hyperparameters from a large candidate space. This work explores the latter hyperparameter optimization (HPO) task for off-policy learning. We empirically show that naively applying an unbiased estimator of the generalization performance as a surrogate objective in HPO can cause an unexpected failure, merely pursuing hyperparameters whose generalization performance is greatly overestimated. We then propose simple and computationally efficient corrections to the typical HPO procedure to deal with the aforementioned issues simultaneously. Empirical investigations demonstrate the effectiveness of our proposed HPO algorithm in situations where the typical procedure fails severely.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualDecision MakingHyperparameter OptimizationOff-policy evaluationRecommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Batch Reinforcement Learning with Hyperparameter Gradients
We consider the batch reinforcement learning problem where the agent needs to learn only from a fixed batch of data, without further interaction with the environment. In such a scenario, we want to prevent the optimized …
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1Online Hyperparameter Adaptation via Amortized Proximal Optimization
Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof). We present Amortized Proximal Optimization (APO), which …
Hyperparameter Optimization with Neural Network Pruning
Since the deep learning model is highly dependent on hyperparameters, hyperparameter optimization is essential in developing deep learning model-based applications, even if it takes a long time. As service development us…
Bayesian OptimizationDeep LearningHyperparameter OptimizationNetwork PruningWhen Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the…
Reinforcement LearningGRPOformer: Advancing Hyperparameter Optimization via Group Relative Policy Optimization
Hyperparameter optimization (HPO) plays a critical role in improving model performance. Transformer-based HPO methods have shown great potential; however, existing approaches rely heavily on large-scale historical optimi…
Hyperparameter OptimizationReinforcement Learning