Optuna Constrained Tree-Structured Parzen Estimator Is a Joint Density Generalization of c-TPE
Constrained hyperparameter optimization (HPO) is common in practice, yet Optuna's widely used constrained TPE lacks algorithmic analysis. While c-TPE proposes an expected constrained improvement (ECI) approach assuming independence between the objective and constraints, Optuna uses a single joint density over both. We show that Optuna's constrained TPE is joint c-TPE -- the same ECI acquisition function using a joint likelihood. We demonstrate joint c-TPE is invariant to constraint duplication whereas independent c-TPE degrades as the product accumulates duplicated factors. We outline practical tradeoffs between the formulations and directions for future study.
Code (0)
등록된 구현이 없습니다.
Tasks
Hyperparameter OptimizationSimilar Papers 제목 키워드 기반
Neural Networks with A La Carte Selection of Activation Functions
Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of n…
Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently
Tree-structured Parzen estimator (TPE) is a versatile hyperparameter optimization (HPO) method supported by popular HPO tools. Since these HPO tools have been developed in line with the trend of deep learning (DL), the p…
Hyperparameter OptimizationOptuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?
Optimal hyperparameter selection is critical for maximizing neural network performance, especially as models grow in complexity. This work investigates the viability of leveraging large language models (LLMs) for hyperpa…
Computational EfficiencyHyperparameter Optimizationc-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as memory usage, or latency on top of the performance requir…
Bayesian OptimizationHyperparameter OptimizationTree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance
Recent advances in many domains require more and more complicated experiment design. Such complicated experiments often have many parameters, which necessitate parameter tuning. Tree-structured Parzen estimator (TPE), a …
Bayesian OptimizationHyperparameter Optimization