How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance. In this work, we propose a new benchmarking protocol to evaluate both end-to-end efficiency (training a model from scratch without knowing the best hyperparameter) and data-addition training efficiency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data). For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy. A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search. For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift. We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining. Our results show that there is no clear winner across all the tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingGraph MiningSimilar Papers 제목 키워드 기반
Measuring Fairness in Generative Models
Deep generative models have made much progress in improving training stability and quality of generated data. Recently there has been increased interest in the fairness of deep-generated data. Fairness is important in ma…
FairnessTowards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle
Deep Learning has enabled many advances in machine learning applications in the last few years. However, since current Deep Learning algorithms require much energy for computations, there are growing concerns about the a…
Deep LearningScaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation
We present a scalable pipeline for automatically generating high-quality training data for web agents. In particular, a major challenge in identifying high-quality training instances is trajectory evaluation - quantifyin…
UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
Significant research efforts have been made to scale and improve vision-language model (VLM) training approaches. Yet, with an ever-growing number of benchmarks, researchers are tasked with the heavy burden of implementi…
GPULanguage ModelingLanguage ModellingObject Recognition+1How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT
This article evaluates purported progress over the past years in RST discourse parsing. Several studies report a relative error reduction of 24 to 51{\%} on all metrics that authors attribute to the introduction of distr…
AttributeDiscourse Parsing