Adaptive Optimization Algorithms for Machine Learning
Machine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This dissertation investigates adaptivity in machine learning optimizers. The ensuing chapters are dedicated to various facets of adaptivity, including: 1. personalization and user-specific models via personalized loss, 2. provable post-training model adaptations via meta-learning, 3. learning unknown hyperparameters in real time via hyperparameter variance reduction, 4. fast O(1/k^2) global convergence of second-order methods via stepsized Newton method regardless of the initialization and choice basis, 5. fast and scalable second-order methods via low-dimensional updates. This thesis contributes novel insights, introduces new algorithms with improved convergence guarantees, and improves analyses of popular practical algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningSecond-order methodsSimilar Papers 제목 키워드 기반
Faster Adaptive Decentralized Learning Algorithms
Decentralized learning recently has received increasing attention in machine learning due to its advantages in implementation simplicity and system robustness, data privacy. Meanwhile, the adaptive gradient methods show …
Stochastic OptimizationAdaptive First- and Second-Order Algorithms for Large-Scale Machine Learning
In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic…
BIG-bench Machine LearningDeep LearningStochastic OptimizationFaster Adaptive Momentum-Based Federated Methods for Distributed Composition Optimization
Federated Learning is a popular distributed learning paradigm in machine learning. Meanwhile, composition optimization is an effective hierarchical learning model, which appears in many machine learning applications such…
Federated LearningMeta-LearningAdaptive versus Standard Descent Methods and Robustness Against Adversarial Examples
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. In this paper we…
FADAS: Towards Federated Adaptive Asynchronous Optimization
Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. While the SGD-based FL algorithms have demonstrated considerable success in the past, there is a growing …
Federated LearningPrivacy Preserving