Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers
Automated hyperparameter optimization (HPO) has gained great popularity and is an important ingredient of most automated machine learning frameworks. The process of designing HPO algorithms, however, is still an unsystematic and manual process: Limitations of prior work are identified and the improvements proposed are -- even though guided by expert knowledge -- still somewhat arbitrary. This rarely allows for gaining a holistic understanding of which algorithmic components are driving performance, and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multifidelity HPO (MF-HPO): First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to common HPO algorithms, and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration, in some ways simpler than established methods, performs very well as long as some critical configuration parameters have the right value.
Code (1)
Tasks
Bayesian OptimizationHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
From Performance to Understanding: A Vision for Explainable Automated Algorithm Design
Automated algorithm design is entering a new phase: Large Language Models can now generate full optimisation (meta)heuristics, explore vast design spaces and adapt through iterative feedback. Yet this rapid progress is l…
o-MEGA: Optimized Methods for Explanation Generation and Analysis
The proliferation of transformer-based language models has revolutionized NLP domain while simultaneously introduced significant challenges regarding model transparency and trustworthiness. The complexity of achieving ex…
Hyperparameter OptimizationExplanation GenerationImproving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution
Despite all the benefits of automated hyperparameter optimization (HPO), most modern HPO algorithms are black-boxes themselves. This makes it difficult to understand the decision process which leads to the selected confi…
Bayesian OptimizationHyperparameter OptimizationInterpretable Machine LearningAutoEP: LLMs-Driven Automation of Hyperparameter Evolution for Metaheuristic Algorithms
Dynamically configuring algorithm hyperparameters is a fundamental challenge in computational intelligence. While learning-based methods offer automation, they suffer from prohibitive sample complexity and poor generaliz…
Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting
Sensors in cyber-physical systems often capture interconnected processes and thus emit correlated time series (CTS), the forecasting of which enables important applications. The key to successful CTS forecasting is to un…
Correlated Time Series ForecastingTime SeriesTime Series AnalysisTime Series Forecasting