Trivial or Impossible --- dichotomous data difficulty masks model differences (on ImageNet and beyond)
`The power of a generalization system follows directly from its biases'' (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides. In particular, we ask: what makes one model decide differently from another? In a meticulously controlled setting, we find that (1.) irrespective of the network architecture or objective (e.g.\ self-supervised, semi-supervised, vision transformers, recurrent models) all models end up with a similar decision boundary. (2.) To understand these findings, we analysed model decisions on the ImageNet validation set from epoch to epoch and image by image. We find that the ImageNet validation set, among others, suffers from dichotomous data difficulty (DDD): For the range of investigated models and their accuracies, it is dominated by 46.0% trivial'' and 11.5% impossible'' images (beyond label errors). Only 42.5% of the images could possibly be responsible for the differences between two models' decision boundaries. (3.) Only removing the impossible'' and trivial'' images allows us to see pronounced differences between models. (4.) Humans are highly accurate at predicting which images are trivial'' and `impossible'' for CNNs (81.4%). This implies that in future comparisons of brains, machines and behaviour, much may be gained from investigating the decisive role of images and the distribution of their difficulties.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasSimilar Papers 제목 키워드 기반
Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
"The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems -- but to what degree have we understood how their inductive bias influe…
Inductive BiasImageNet suffers from dichotomous data difficulty
"The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influen…
Inductive BiasDeep Learning Segmentation of Spiral Arms and Bars
We present the first deep learning model for segmenting galactic spiral arms and bars. In a blinded assessment by expert astronomers, our predicted spiral arm masks are preferred over both current automated methods (99% …
Deep LearningThe Two-Hump Problem: Bridging the Difficulty Gap in Mathematical Reinforcement Learning
Mathematical search problems present a unique challenge for Reinforcement Learning (RL) due to vast search spaces and sparse rewards. In previous works, the Andrews-Curtis (AC) conjecture was established as an illustrati…
Reinforcement LearningPromoting Segment Anything Model towards Highly Accurate Dichotomous Image Segmentation
The Segment Anything Model (SAM) represents a significant breakthrough into foundation models for computer vision, providing a large-scale image segmentation model. However, despite SAM's zero-shot performance, its segme…
Dichotomous Image SegmentationImage SegmentationSegmentationSemantic Segmentation