VASSO: Variance Suppression for Sharpness-Aware Minimization
Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, where neighborhoods of flat minima' heighten generalization ability, SAM seeks flat valleys' by minimizing the maximum loss provoked by an adversarial perturbation within the neighborhood. Although critical to account for sharpness of the loss function, in practice SAM suffers from over-friendly adversaries,' which can curtail the outmost level of generalization. To avoid such friendliness,' the present contribution fosters stabilization of adversaries through variance suppression (VASSO). VASSO offers a general approach to provably stabilize adversaries. In particular, when integrating VASSO with SAM, improved generalizability is numerically validated on extensive vision and language tasks. Once applied on top of a computationally efficient SAM variant, VASSO offers a desirable generalization-computation tradeoff.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Enhancing Sharpness-Aware Optimization Through Variance Suppression
Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation. Embracing the geometry of the loss function, where neighborhoods …
Data Augmentationimage-classificationImage ClassificationMachine TranslationA Universal Class of Sharpness-Aware Minimization Algorithms
Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on…
GCSAM: Gradient Centralized Sharpness Aware Minimization
The generalization performance of deep neural networks (DNNs) is a critical factor in achieving robust model behavior on unseen data. Recent studies have highlighted the importance of sharpness-based measures in promotin…
Computational EfficiencyModel Generalization: A Sharpness Aware Optimization Perspective
Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimization (ASAM) aim to improve the model generalization. And in this project, we proposed three experiments to valid their generalization from the sharp…
modelvalidFedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learning
Sharpness-aware minimization (SAM) is an effective method for improving the generalization of federated learning (FL) by steering local training toward flat minima. Under data heterogeneity, however, device-side SAM sear…
Federated Learning