Adaptive Optimizers with Sparse Group Lasso
We develop a novel framework that adds the regularizers to a family of adaptive optimizers in deep learning, such as MOMENTUM, ADAGRAD, ADAM, AMSGRAD, ADAHESSIAN, and create a new class of optimizers, which are named GROUP MOMENTUM, GROUP ADAGRAD, GROUP ADAM, GROUP AMSGRAD and GROUP ADAHESSIAN, etc., accordingly. We establish theoretically proven convergence guarantees in the stochastic convex settings, based on primal-dual methods. We evaluate the regularized effect of our new optimizers on three large scale real-world ad click datasets with state-of-the-art deep learning models. The experimental results reveal that compared with the traditional methods adding regularization terms to the loss functions, not only can the dimensions of features be effectively and efficiently reduced, the performance of the models could also get improved. Furthermore, in comparison to the cases with no regularization terms, our methods can achieve extremely sparsity with highly competitive or significantly better performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction
We develop a novel framework that adds the regularizers of the sparse group lasso to a family of adaptive optimizers in deep learning, such as Momentum, Adagrad, Adam, AMSGrad, AdaHessian, and create a new class of optim…
Click-Through Rate PredictionDual feature reduction for the sparse-group lasso and its adaptive variant
The sparse-group lasso performs both variable and group selection, making simultaneous use of the strengths of the lasso and group lasso. It has found widespread use in genetics, a field that regularly involves the analy…
Adaptive Class Preserving Representation for Image Classification
In linear representation-based image classification, an unlabeled sample is represented by the entire training set. To obtain a stable and discriminative solution, regularization on the vector of representation coefficie…
ClassificationGeneral Classificationimage-classificationImage Classification+1Efficient Clustering of Correlated Variables and Variable Selection in High-Dimensional Linear Models
In this paper, we introduce Adaptive Cluster Lasso(ACL) method for variable selection in high dimensional sparse regression models with strongly correlated variables. To handle correlated variables, the concept of cluste…
ClusteringVariable SelectionAdaptive Group Lasso Neural Network Models for Functions of Few Variables and Time-Dependent Data
In this paper, we propose an adaptive group Lasso deep neural network for high-dimensional function approximation where input data are generated from a dynamical system and the target function depends on few active varia…