On the Variance of Neural Network Training with respect to Test Sets and Distributions
Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. In this work we present the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in performance on the underlying test-distributions from which their test-sets are sampled. (2) We show that these trainings make approximately independent errors on their test-sets. That is, the event that a trained network makes an error on one particular example does not affect its chances of making errors on other examples, relative to their average rates over repeated runs of training with the same hyperparameters. (3) We prove that the variance of neural network trainings on their test-sets is a downstream consequence of the class-calibration property discovered by Jiang et al. (2021). Our analysis yields a simple formula which accurately predicts variance for the binary classification case. (4) We conduct preliminary studies of data augmentation, learning rate, finetuning instability and distribution-shift through the lens of variance between runs.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationData AugmentationSimilar Papers 제목 키워드 기반
Task-Adaptive Few-shot Node Classification
Node classification is of great importance among various graph mining tasks. In practice, real-world graphs generally follow the long-tail distribution, where a large number of classes only consist of limited labeled nod…
ClassificationFew-Shot LearningGraph MiningNode ClassificationLearnable Distribution Calibration for Few-Shot Class-Incremental Learning
Few-shot class-incremental learning (FSCIL) faces challenges of memorizing old class distributions and estimating new class distributions given few training samples. In this study, we propose a learnable distribution cal…
class-incremental learningClass Incremental LearningFew-Shot Class-Incremental LearningFew-Shot Learning+2Domain Generalisation via Risk Distribution Matching
We propose a novel approach for domain generalisation (DG) leveraging risk distributions to characterise domains, thereby achieving domain invariance. In our findings, risk distributions effectively highlight differences…
A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs
Steerable convolutional neural networks (SCNNs) enhance task performance by modelling geometric symmetries through equivariance constraints on weights. Yet, unknown or varying symmetries can lead to overconstrained weigh…
Domain Generalization via Domain-based Covariance Minimization
Researchers have been facing a difficult problem that data generation mechanisms could be influenced by internal or external factors leading to the training and test data with quite different distributions, consequently …
Dimensionality ReductionDomain Generalization