On the Algorithmic Stability and Generalization of Adaptive Optimization Methods
Despite their popularity in deep learning and machine learning in general, the theoretical properties of adaptive optimizers such as Adagrad, RMSProp, Adam or AdamW are not yet fully understood. In this paper, we develop a novel framework to study the stability and generalization of these optimization methods. Based on this framework, we show provable guarantees about such properties that depend heavily on a single parameter $\beta_2$. Our empirical experiments support our claims and provide practical insights into the stability and generalization properties of adaptive optimization methods.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problems
Stochastic optimization has found wide applications in minimizing objective functions in machine learning, which motivates a lot of theoretical studies to understand its practical success. Most of existing studies focus …
Stochastic OptimizationUnderstanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization
Federated Learning (FL) is a distributed learning approach that trains neural networks across multiple devices while keeping their local data private. However, FL often faces challenges due to data heterogeneity, leading…
Federated LearningOn the Stability and Generalization of First-order Bilevel Minimax Optimization
Bilevel optimization and bilevel minimax optimization have recently emerged as unifying frameworks for a range of machine-learning tasks, including hyperparameter optimization and reinforcement learning. The existing lit…
Hyperparameter OptimizationReinforcement LearningBilevel OptimizationExploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability
Stochastic optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning. Although various algorithms have been extensively studied for AUPRC optimization, the generalization…
Generalization BoundsImage RetrievalRetrievalStochastic OptimizationAlgorithmic Stability and Uniform Generalization
One of the central questions in statistical learning theory is to determine the conditions under which agents can learn from experience. This includes the necessary and sufficient conditions for generalization from a giv…
Dimensionality ReductionLearning Theory