Fast Optimizer Benchmark
In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning. The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities. FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs. The modular design enables integration into custom pipelines, using it simply as a collection of tasks. We showcase an optimizer comparison as a usage example of our tool. FOB can be found on GitHub: https://github.com/automl/FOB.
Code (1)
Tasks
AutoMLGraph LearningHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months
We analyze VeLO (versatile learned optimizer), the largest scale attempt to train a general purpose "foundational" optimizer to date. VeLO was trained on thousands of machine learning tasks using over 4000 TPU months wit…
Radiotherapy Dosimetry: A Review on Open-Source Optimizer
Radiotherapy dosimetry plays a crucial role in optimizing treatment plans for cancer patients. In this study, we investigate the performance of a dozen standard state-of-the-art open-source optimizers for radiotherapy do…
AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization
Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implement…
Optimizing the Optimizer: Language Models Discover Faster Molecular Relaxation
Geometry optimization is a major cost in many quantum-chemical workflows: each optimization step requires one force evaluation, and at the density-functional level that evaluation dominates the wall time. Research in thi…
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normalization to the update rules of Heavy-ball momentum and Adam, respec…
Image ClassificationOpen-Ended Question AnsweringQuestion AnsweringStochastic Optimization