Fast Optimizer Benchmark
In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning. The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities. FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs. The modular design enables integration into custom pipelines, using it simply as a collection of tasks. We showcase an optimizer comparison as a usage example of our tool. FOB can be found on GitHub: https://github.com/automl/FOB.
Code (1)
Tasks
AutoMLGraph LearningHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months
We analyze VeLO (versatile learned optimizer), the largest scale attempt to train a general purpose "foundational" optimizer to date. VeLO was trained on thousands of machine learning tasks using over 4000 TPU months wit…
Radiotherapy Dosimetry: A Review on Open-Source Optimizer
Radiotherapy dosimetry plays a crucial role in optimizing treatment plans for cancer patients. In this study, we investigate the performance of a dozen standard state-of-the-art open-source optimizers for radiotherapy do…
AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization
Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implement…
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normalization to the update rules of Heavy-ball momentum and Adam, respec…
Image ClassificationOpen-Ended Question AnsweringQuestion AnsweringStochastic OptimizationEffective Quantization of Muon Optimizer States
The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and better computational efficiency over AdamW in LLM pre-training. However, the memory overhead of maintaining high-precision …
Computational EfficiencyInstruction Following