Low-Cost Recurrent Neural Network Expected Performance Evaluation
Recurrent neural networks are a powerful tool, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. Varied strategies have been proposed to tackle this issue. However, most of them are still impractical because of the time/resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples of the weights. We empirically validate our proposal using three use cases. The results suggest that this is a promising alternative to reduce the cost of exploration for hyper-parameter optimization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs
Bayesian optimization (BO) is a sample-efficient approach to optimizing costly-to-evaluate black-box functions. Most BO methods ignore how evaluation costs may vary over the optimization domain. However, these costs can …
Bayesian OptimizationRestricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
Increasing the capacity of recurrent neural networks (RNN) usually involves augmenting the size of the hidden layer, with significant increase of computational cost. Recurrent neural tensor networks (RNTN) increase capac…
Language ModelingLanguage ModellingTensor NetworksAdjusted Expected Improvement for Cumulative Regret Minimization in Noisy Bayesian Optimization
The expected improvement (EI) is one of the most popular acquisition functions for Bayesian optimization (BO) and has demonstrated good empirical performances in many applications for the minimization of simple regret. H…
Bayesian OptimizationThompson SamplingRandom Error Sampling-based Recurrent Neural Network Architecture Optimization
Recurrent neural networks are good at solving prediction problems. However, finding a network that suits a problem is quite hard because their performance is strongly affected by their architecture configuration. Automat…
Hyperparameter OptimizationCost-aware Stopping for Bayesian Optimization
In automated machine learning, scientific discovery, and other applications of Bayesian optimization, deciding when to stop evaluating expensive black-box functions in a cost-aware manner is an important but underexplore…
Hyperparameter Optimization