ImageNet suffers from dichotomous data difficulty
"The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides. In particular, we ask: what makes one model decide differently from another? In a meticulously controlled setting, we find that (1.) irrespective of the network architecture or objective (e.g. self-supervised, semi-supervised, vision transformers, recurrent models) all models end up with a similar decision boundary. (2.) To understand these findings, we analysed model decisions on the ImageNet validation set from epoch to epoch and image by image. We find that the ImageNet validation set suffers from dichotomous data difficulty (DDD): For the range of investigated models and their accuracies, it is dominated by 46.3% "trivial" and 11.3% "impossible" images. Only 42.4% of the images are responsible for the differences between two models' decision boundaries. The impossible images are not driven by label errors. (3.) Finally, humans are highly accurate at predicting which images are "trivial" and "impossible" for CNNs (81.4%). Taken together, it appears that ImageNet suffers from dichotomous data difficulty. This implies that in future comparisons of brains, machines and behaviour, much may be gained from investigating the decisive role of images and the distribution of their difficulties.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasSimilar Papers 제목 키워드 기반
Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
"The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems -- but to what degree have we understood how their inductive bias influe…
Inductive BiasTrivial or Impossible --- dichotomous data difficulty masks model differences (on ImageNet and beyond)
``The power of a generalization system follows directly from its biases'' (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influ…
Inductive BiasPareto efficient combinatorial auctions: dichotomous preferences without quasilinearity
We consider a combinatorial auction model where preferences of agents over bundles of objects and payments need not be quasilinear. However, we restrict the preferences of agents to be dichotomous. An agent with dichotom…
SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
Adversarial training (AT) has become a popular choice for training robust networks. However, it tends to sacrifice clean accuracy heavily in favor of robustness and suffers from a large generalization error. To address t…
Adversarial RobustnessA study on the invariance in security whatever the dimension of images for the steganalysis by deep-learning
In this paper, we study the performance invariance of convolutional neural networks when confronted with variable image sizes in the context of a more "wild steganalysis". First, we propose two algorithms and definitions…
Steganalysis