LALR: Theoretical and Experimental validation of Lipschitz Adaptive Learning Rate in Regression and Neural Networks
We propose a theoretical framework for an adaptive learning rate policy for the Mean Absolute Error loss function and Quantile loss function and evaluate its effectiveness for regression tasks. The framework is based on the theory of Lipschitz continuity, specifically utilizing the relationship between learning rate and Lipschitz constant of the loss function. Based on experimentation, we have found that the adaptive learning rate policy enables up to 20x faster convergence compared to a constant learning rate policy.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionSimilar Papers 제목 키워드 기반
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data
Federated learning is an emerging distributed machine learning method, enables a large number of clients to train a model without exchanging their local data. The time cost of communication is an essential bottleneck in …
Federated LearningSchedulingEstimation and Applications of Quantiles in Deep Binary Classification
Quantile regression, based on check loss, is a widely used inferential paradigm in Econometrics and Statistics. The conditional quantiles provide a robust alternative to classical conditional means, and also allow uncert…
Binary ClassificationClassificationEconometricsGeneral Classification+3Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness
Convergence and convergence rate analyses of adaptive methods, such as Adaptive Moment Estimation (Adam) and its variants, have been widely studied for nonconvex optimization. The analyses are based on assumptions that t…
A Control Theoretical Adaptive Human Pilot Model: Theory and Experimental Validation
This paper proposes an adaptive human pilot model that is able to mimic the crossover model in the presence of uncertainties. The proposed structure is based on the model reference adaptive control, and the adaptive laws…
Parsimonious Computing: A Minority Training Regime for Effective Prediction in Large Microarray Expression Data Sets
Rigorous mathematical investigation of learning rates used in back-propagation in shallow neural networks has become a necessity. This is because experimental evidence needs to be endorsed by a theoretical background. Su…