Hierarchical Simplicity Bias of Neural Networks
Neural networks often exhibit simplicity bias, favoring simpler features over more complex ones, even when both are equally predictive. We introduce a novel method called imbalanced label coupling to explore and extend this simplicity bias across multiple hierarchical levels. Our approach demonstrates that trained networks sequentially consider features of increasing complexity based on their correlation with labels in the training set, regardless of their actual predictive power. For example, in CIFAR-10, simple spurious features can cause misclassifications where most cats are predicted as dogs and most trucks as automobiles. We empirically show that last-layer retraining with target data distribution \citep{kirichenko2022last} is insufficient to fully recover core features when spurious features perfectly correlate with target labels in our synthetic datasets. Our findings deepen the understanding of the implicit biases inherent in neural networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Simplicity bias in the parameter-function map of deep neural networks
The idea that neural networks may exhibit a bias towards simplicity has a long history. Simplicity bias provides a way to quantify this intuition. It predicts, for a broad class of input-output maps which can describe m…
Simplicity Bias of Transformers to Learn Low Sensitivity Functions
Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of the inductive biases that they have and how those biases are different from other neural network architectures rema…
SensitivityDo We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
Neural architectures tend to fit their data with relatively simple functions. This "simplicity bias" is widely regarded as key to their success. This paper explores the limits of this principle. Building on recent findin…
image-classificationImage ClassificationMitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness
Neural networks (NNs) are known to exhibit simplicity bias where they tend to prefer learning 'simple' features over more 'complex' ones, even when the latter may be more informative. Simplicity bias can lead to the mode…
FairnessSimplicity bias, algorithmic probability, and the random logistic map
Simplicity bias is an intriguing phenomenon prevalent in various input-output maps, characterized by a preference for simpler, more regular, or symmetric outputs. Notably, these maps typically feature high-probability ou…