Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization

2023-03-03 · CVPR 2023 1 · Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, Peng Cui

Recently, flat minima are proven to be effective for improving generalization and sharpness-aware minimization (SAM) achieves state-of-the-art performance. Yet the current definition of flatness discussed in SAM and its follow-ups are limited to the zeroth-order flatness (i.e., the worst-case loss within a perturbation radius). We show that the zeroth-order flatness can be insufficient to discriminate minima with low generalization error from those with high generalization error both when there is a single minimum or multiple minima within the given perturbation radius. Thus we present first-order flatness, a stronger measure of flatness focusing on the maximal gradient norm within a perturbation radius which bounds both the maximal eigenvalue of Hessian at local minima and the regularization function of SAM. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions. Experimental results show that GAM improves the generalization of models trained with current optimizers such as SGD and AdamW on various datasets and networks. Furthermore, we show that GAM can help SAM find flatter minima and achieve better generalization.

📄 PDF Abstract BibTeX arXiv:2303.03108

Code (1)

xxgege/GAM 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Sharpness-Aware Minimization Sharpness-Aware Minimization, or SAM, is a procedure that improves model generalization by simultaneously minimizing loss value and loss sharpness. SAM functions by…

AdamW AdamW is a stochastic optimization method that modifies the typical implementation of weight decay in Adam, by decoupling [weight…

GAM 설명 없음

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning

2026-02-10 · Yifei Cheng, Xianglin Yang, Guoxia Wang, Chao Huang 외 arxiv

Sharpness-aware minimization (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning. However, its extra parameter perturbation step do…

Efficient Continual Finite-Sum Minimization

2024-06-07 · Ioannis Mavrothalassitis, Stratis Skoulakis, Leello Tadesse Dadi, Volkan Cevher

Given a sequence of functions $f_1,\ldots,f_n$ with $f_i:\mathcal{D}\mapsto \mathbb{R}$, finite-sum minimization seeks a point ${x}^\star \in \mathcal{D}$ minimizing $\sum_{j=1}^n f_j(x)/n$. In this work, we propose a ke…

Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Models

2025-08-14 · Tianxiao Cao, Kyohei Atarashi, Hisashi Kashima arxiv

Sharpness-Aware Minimization (SAM) has been proven to be an effective optimization technique for improving generalization in overparameterized models. While prior works have explored the implicit regularization of SAM in…

parameter-efficient fine-tuning Model Compression

Sharpness-Aware Minimization and the Edge of Stability

2023-09-21 · Philip M. Long, Peter L. Bartlett

Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size $\eta$, the operator norm of the Hessian of the loss grows until it approximately reaches $2/\eta$, af…

GCSAM: Gradient Centralized Sharpness Aware Minimization

2025-01-20 · Mohamed Hassan, Aleksandar Vakanski, Boyu Zhang, Min Xian

The generalization performance of deep neural networks (DNNs) is a critical factor in achieving robust model behavior on unseen data. Recent studies have highlighted the importance of sharpness-based measures in promotin…

Computational Efficiency