Multi-objective Optimization by Learning Space Partitions
In contrast to single-objective optimization (SOO), multi-objective optimization (MOO) requires an optimizer to find the Pareto frontier, a subset of feasible solutions that are not dominated by other feasible solutions. In this paper, we propose LaMOO, a novel multi-objective optimizer that learns a model from observed samples to partition the search space and then focus on promising regions that are likely to contain a subset of the Pareto frontier. The partitioning is based on the dominance number, which measures "how close" a data point is to the Pareto frontier among existing samples. To account for possible partition errors due to limited samples and model mismatch, we leverage Monte Carlo Tree Search (MCTS) to exploit promising regions while exploring suboptimal regions that may turn out to contain good solutions later. Theoretically, we prove the efficacy of learning space partitioning via LaMOO under certain assumptions. Empirically, on the HyperVolume (HV) benchmark, a popular MOO metric, LaMOO substantially outperforms strong baselines on multiple real-world MOO tasks, by up to 225% in sample efficiency for neural architecture search on Nasbench201, and up to 10% for molecular design.
Code (1)
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
Multi-Objective Neural Architecture Search by Learning Search Space Partitions
Deploying deep learning models requires taking into consideration neural network metrics such as model size, inference latency, and #FLOPs, aside from inference accuracy. This results in deep learning model designers lev…
Bayesian OptimizationGPUNeural Architecture SearchGaussianPSL: Soft partitioning for complex PSL problem
Many practical applications of multi-objective optimization (MOO), including engineering design, autonomous systems, and machine learning, often yield complex Pareto frontiers (e.g., discontinuous, degenerate, or non-con…
Multi-Objective Bayesian Optimization for Model Merging
Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are …
NeuroCUT: A Neural Approach for Robust Graph Partitioning
Graph partitioning aims to divide a graph into disjoint subsets while optimizing a specific partitioning objective. The majority of formulations related to graph partitioning exhibit NP-hardness due to their combinatoria…
Graph Neural Networkgraph partitioningHyperparameter Optimization through Neural Network Partitioning
Well-tuned hyperparameters are crucial for obtaining good generalization behavior in neural networks. They can enforce appropriate inductive biases, regularize the model and improve performance -- especially in the prese…
Federated LearningHyperparameter Optimization