Deep $k$-NN Label Smoothing Improves Reproducibility of Neural Network Predictions
Training modern neural networks is an inherently noisy process that can lead to high \emph{prediction churn}-- disagreements between re-trainings of the same model due to factors such as randomization in the parameter initialization and mini-batches-- even when the trained models all attain high accuracies. Such prediction churn can be very undesirable in practice. In this paper, we present several baselines for reducing churn and show that utilizing the $k$-NN predictions to smooth the labels results in a new and principled method that outperforms the baselines on churn while improving accuracy on a variety of benchmark classification tasks and model architectures.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionSimilar Papers 제목 키워드 기반
Label Smoothing++: Enhanced Label Regularization for Training Neural Networks
Training neural networks with one-hot target labels often results in overconfidence and overfitting. Label smoothing addresses this issue by perturbing the one-hot target labels by adding a uniform probability vector to …
When Does Label Smoothing Help?
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Sm…
image-classificationImage ClassificationKnowledge Distillationspeech-recognition+2In and Out-of-Domain Text Adversarial Robustness via Label Smoothing
Recently it has been shown that state-of-the-art NLP models are vulnerable to adversarial attacks, where the predictions of a model can be drastically altered by slight modifications to the input (such as synonym substit…
Adversarial RobustnessFedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration
Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods. This manifests itself as either overconfident incorrect predic…
Improving Predictive Confidence in Medical Imaging via Online Label Smoothing
Deep learning models, especially convolutional neural networks, have achieved impressive results in medical image classification. However, these models often produce overconfident predictions, which can undermine their r…
Medical Image ClassificationRepresentation LearningKnowledge Distillation