Start Small, Think Big: On Hyperparameter Optimization for Large-Scale Knowledge Graph Embeddings
Knowledge graph embedding (KGE) models are an effective and popular approach to represent and reason with multi-relational data. Prior studies have shown that KGE models are sensitive to hyperparameter settings, however, and that suitable choices are dataset-dependent. In this paper, we explore hyperparameter optimization (HPO) for very large knowledge graphs, where the cost of evaluating individual hyperparameter configurations is excessive. Prior studies often avoided this cost by using various heuristics; e.g., by training on a subgraph or by using fewer epochs. We systematically discuss and evaluate the quality and cost savings of such heuristics and other low-cost approximation techniques. Based on our findings, we introduce GraSH, an efficient multi-fidelity HPO algorithm for large-scale KGEs that combines both graph and epoch reduction techniques and runs in multiple rounds of increasing fidelities. We conducted an experimental study and found that GraSH obtains state-of-the-art results on large graphs at a low cost (three complete training runs in total).
Code (2)
Tasks
Graph EmbeddingHyperparameter OptimizationKnowledge Graph EmbeddingKnowledge Graph EmbeddingsKnowledge GraphsLink PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ExperienceThinking: Constrained Hyperparameter Optimization based on Knowledge and Pruning
Machine learning algorithms are very sensitive to the hyperparameters, and their evaluations are generally expensive. Users desperately need intelligent methods to quickly optimize hyperparameter settings according to kn…
BIG-bench Machine LearningHyperparameter OptimizationNeural Architecture SearchRethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization
Effectively representing heterogeneous tabular datasets for meta-learning purposes remains an open problem. Previous approaches rely on predefined meta-features, for example, statistical measures or landmarkers. The emer…
AttributeHyperparameter OptimizationMeta-LearningRepresentation LearningWarmstarting for Scaling Language Models
Scaling model sizes to scale performance has worked remarkably well for the current large language models paradigm. The research and empirical findings of various scaling studies led to novel scaling results and laws tha…
Learning to Warm-Start Bayesian Hyperparameter Optimization
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a lo…
Bayesian Optimizationglobal-optimizationHyperparameter OptimizationMetric LearningImportance-Aware Scheduling for High-Dimensional Hyperparameter Optimization
Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across …
Hyperparameter Optimization