Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective function by adding a regularization term on the learning rate, and casts the joint updating process of parameters and learning rates into a maxmin problem. Given any regularization term, our approach facilitates the generation of practical algorithms. When \textit{Meta-Regularization} takes the $\varphi$-divergence as a regularizer, the resulting algorithms exhibit comparable theoretical convergence performance with other first-order gradient-based algorithms. Furthermore, we theoretically prove that some well-designed regularizers can improve the convergence performance under the strong-convexity condition of the objective function. Numerical experiments on benchmark problems demonstrate the effectiveness of algorithms derived from some common $\varphi$-divergence in full batch as well as online learning settings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continu…
Class Incremental LearningContinual LearningAdaptive Meta-learner via Gradient Similarity for Few-shot Text Classification
Few-shot text classification aims to classify the text under the few-shot scenario. Most of the previous methods adopt optimization-based meta learning to obtain task distribution. However, due to the neglect of matching…
Few-Shot Text ClassificationMeta-Learningtext-classificationText ClassificationHyper-Regularization: An Adaptive Choice for the Learning Rate in Gradient Descent
We present a novel approach for adaptively selecting the learning rate in gradient descent methods. Specifically, we impose a regularization term on the learning rate via a generalized distance, and cast the joint updat…
Adaptive regularization for Lasso models in the context of non-stationary data streams
Large scale, streaming datasets are ubiquitous in modern machine learning. Streaming algorithms must be scalable, amenable to incremental training and robust to the presence of non-stationarity. In this work consider the…
Cooperative Meta-Learning with Gradient Augmentation
Model agnostic meta-learning (MAML) is one of the most widely used gradient-based meta-learning, consisting of two optimization loops: an inner loop and outer loop. MAML learns the new task from meta-initialization param…
Few-Shot Image Classificationimage-classificationImage ClassificationMeta-Learning+1