When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statistics
Although a plethora of architectural variants for deep classification has been introduced over time, recent works have found empirical evidence towards similarities in their training process. It has been hypothesized that neural networks converge not only to similar representations, but also exhibit a notion of empirical agreement on which data instances are learned first. Following in the latter works$'$ footsteps, we define a metric to quantify the relationship between such classification agreement over time, and posit that the agreement phenomenon can be mapped to core statistics of the investigated dataset. We empirically corroborate this hypothesis across the CIFAR10, Pascal, ImageNet and KTH-TIPS2 datasets. Our findings indicate that agreement seems to be independent of specific architectures, training hyper-parameters or labels, albeit follows an ordering according to image statistics.
Code (1)
Similar Papers 제목 키워드 기반
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods
A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretabil…
DiagnosticMitigating Spurious Correlations via Disagreement Probability
Models trained with empirical risk minimization (ERM) are prone to be biased towards spurious correlations between target labels and bias attributes, which leads to poor performance on data groups lacking spurious correl…
Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Recently, Miller et al. showed that a model's in-distribution (ID) accuracy has a strong linear correlation with its out-of-distribution (OOD) accuracy on several OOD benchmarks -- a phenomenon they dubbed ''accuracy-on-…
Model SelectionUnder the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
How do neural language models keep track of number agreement between subject and verb? We show that `diagnostic classifiers', trained to predict number from the internal states of a language model, provide a detailed und…
DiagnosticLanguage ModelingLanguage ModellingFunzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives
In this work, we evaluate annotator disagreement in Word-in-Context (WiC) tasks exploring the relationship between contextual meaning and disagreement as part of the CoMeDi shared task competition. While prior studies ha…
Sentence