High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
We investigate the relationship between the frequency spectrum of image data and the generalization behavior of convolutional neural networks (CNN). We first notice CNN's ability in capturing the high-frequency components of images. These high-frequency components are almost imperceptible to a human. Thus the observation leads to multiple hypotheses that are related to the generalization behaviors of CNN, including a potential explanation for adversarial examples, a discussion of CNN's trade-off between robustness and accuracy, and some evidence in understanding training heuristics.
Code (1)
Tasks
Adversarial AttackVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
We investigate the relationship between the frequency spectrum of image data and the generalization behavior of convolutional neural networks (CNN). We first notice CNN's ability in capturing the high-frequency component…
Vocal Bursts Intensity PredictionEquivalence between Geometric Frequency and Lagrange Derivative
The paper shows the equivalence between the geometric frequency of an electric quantity, namely, voltage and current, and the Lagrange derivative of a stream-line of a fluid. The geometric frequency is a concept recently…
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose…
Phase Matching for Out-of-Distribution Generalization
The Fourier transform, an explicit decomposition method for visual signals, has been employed to explain the out-of-distribution generalization behaviors of Deep Neural Networks (DNNs). Previous studies indicate that the…
Contrastive LearningDomain GeneralizationOut-of-Distribution GeneralizationRepresentation LearningDC is all you need: describing ReLU from a signal processing standpoint
Non-linear activation functions are crucial in Convolutional Neural Networks. However, until now they have not been well described in the frequency domain. In this work, we study the spectral behavior of ReLU, a popular …
All