Towards Explaining Hyperparameter Optimization via Partial Dependence Plots
Automated hyperparameter optimization (HPO) can support practitioners to obtain peak performance in machine learning models. However, there is often a lack of valuable insights into the effects of different hyperparameters on the final model performance. This lack of comprehensibility and transparency makes it difficult to trust and understand the automated HPO process and its results. We suggest using interpretable machine learning (IML) to gain insights from the experimental data obtained during HPO and especially discuss the popular case of Bayesian optimization (BO). BO tends to focus on promising regions with potential high-performance configurations and thus induces a sampling bias. Hence, many IML techniques, like Partial Dependence Plots (PDP), carry the risk of generating biased interpretations. By leveraging the posterior uncertainty of the BO surrogate model, we introduce a variant of the PDP with estimated confidence bands. In addition, we propose to partition the hyperparameter space to obtain more confident and reliable PDPs in relevant sub-regions. In an experimental study, we provide quantitative evidence for the increased quality of the PDPs within sub-regions.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationBIG-bench Machine LearningHyperparameter OptimizationInterpretable Machine LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Explaining Hyperparameter Optimization via Partial Dependence Plots
Automated hyperparameter optimization (HPO) can support practitioners to obtain peak performance in machine learning models. However, there is often a lack of valuable insights into the effects of different hyperparamete…
Bayesian OptimizationBIG-bench Machine LearningHyperparameter OptimizationInterpretable Machine LearningModel Interpretation: A Unified Derivative-based Framework for Nonparametric Regression and Supervised Machine Learning
Interpreting a nonparametric regression model with many predictors is known to be a challenging problem. There has been renewed interest in this topic due to the extensive use of machine learning algorithms and the diffi…
BIG-bench Machine LearningregressionImproving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution
Despite all the benefits of automated hyperparameter optimization (HPO), most modern HPO algorithms are black-boxes themselves. This makes it difficult to understand the decision process which leads to the selected confi…
Bayesian OptimizationHyperparameter OptimizationInterpretable Machine LearningFooling Partial Dependence via Data Poisoning
Many methods have been developed to understand complex predictive models and high expectations are placed on post-hoc model explainability. It turns out that such explanations are not robust nor trustworthy, and they can…
Data PoisoningAutomated Dependence Plots
In practical applications of machine learning, it is necessary to look beyond standard metrics such as test accuracy in order to validate various qualitative properties of a model. Partial dependence plots (PDP), includi…
Bias DetectionModel SelectionSelection bias