Normalization Layers Are All That Sharpness-Aware Minimization Needs

2023-06-07 · NeurIPS 2023 11 · Maximilian Mueller, Tiffany Vlaar, David Rolnick, Matthias Hein

Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings. In this work we show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.This finding generalizes to different SAM variants and both ResNet (Batch Normalization) and Vision Transformer (Layer Normalization) architectures. We consider alternative sparse perturbation approaches and find that these do not achieve similar performance enhancement at such extreme sparsity levels, showing that this behaviour is unique to the normalization layers. Although our findings reaffirm the effectiveness of SAM in improving generalization performance, they cast doubt on whether this is solely caused by reduced sharpness.

📄 PDF Abstract BibTeX arXiv:2306.04226

Code (1)

mueller-mp/sam-on 공식 구현 pytorch

Tasks

All

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Average Pooling 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

Max Pooling Max Pooling is a pooling operation that calculates the maximum value for patches of a feature map, and uses it to create a downsampled (pooled) feature map. It is usually…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

The Crucial Role of Normalization in Sharpness-Aware Minimization

2023-05-24 · NeurIPS 2023 11

Sharpness-Aware Minimization (SAM) is a recently proposed gradient-based optimizer (Foret et al., ICLR 2021) that greatly improves the prediction performance of deep neural networks. Consequently, there has been a surge …

SAMSON: Sharpness-Aware Minimization Scaled by Outlier Normalization for Improving DNN Generalization and Robustness

2022-11-18 · Gonçalo Mordido, Sébastien Henwood, Sarath Chandar, François Leduc-Primeau

Energy-efficient deep neural network (DNN) accelerators are prone to non-idealities that degrade DNN performance at inference time. To mitigate such degradation, existing methods typically add perturbations to the DNN we…

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy

2024-01-14 · Chengli Tan, Jiangshe Zhang, Junmin Liu, Yicheng Wang 외

Recently, sharpness-aware minimization (SAM) has attracted much attention because of its surprising effectiveness in improving generalization performance. However, compared to stochastic gradient descent (SGD), it is mor…

Learning Theory

Sharpness-Aware Minimization with Z-Score Gradient Filtering for Neural Networks

2025-05-05 · Juyoung Yun

Sharpness-Aware Minimization (SAM) improves neural network generalization by optimizing the worst-case loss within a neighborhood of parameters, yet it perturbs parameters using the entire gradient vector, including comp…

Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfaces

2024-12-02 · Tianwang Jia, Lubin Meng, Siyang Li, Jiajing Liu 외

Training an accurate classifier for EEG-based brain-computer interface (BCI) requires EEG data from a large number of users, whereas protecting their data privacy is a critical consideration. Federated learning (FL) is a…

Brain Computer Interface EEG Federated Learning Motor Imagery+1