Retraining with Predicted Hard Labels Provably Increases Model Accuracy
The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable binary classification setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with local label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at no extra privacy cost; we call this consensus-based retraining. As an example, when training ResNet-18 on CIFAR-100 with $\epsilon=3$ label DP, we obtain more than 6% improvement in accuracy with consensus-based retraining.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationSimilar Papers 제목 키워드 기반
MultiGuard: Provably Robust Multi-label Classification against Adversarial Examples
Multi-label classification, which predicts a set of labels for an input, has many applications. However, multiple recent studies showed that multi-label classification is vulnerable to adversarial examples. In particular…
ClassificationMulti-class ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONIsotonic Data Augmentation for Knowledge Distillation
Knowledge distillation uses both real hard labels and soft labels predicted by teacher models as supervision. Intuitively, we expect the soft labels and hard labels to be concordant w.r.t. their orders of probabilities. …
AttributeData AugmentationGPUKnowledge Distillation+1Provably Secure Federated Learning against Malicious Clients
Federated learning enables clients to collaboratively learn a shared global model without sharing their local training data with a cloud server. However, malicious clients can corrupt the global model to predict incorrec…
Activity RecognitionFederated LearningHuman Activity RecognitionTowards Counteracting Adversarial Perturbations to Resist Adversarial Examples
Studies show that neural networks are susceptible to adversarial attacks. This exposes a potential threat to neural network-based artificial intelligence systems. We observe that the probability of the correct result out…
Improving Transformation-based Defenses against Adversarial Examples with First-order Perturbations
Deep neural networks have been successfully applied in various machine learning tasks. However, studies show that neural networks are susceptible to adversarial attacks. This exposes a potential threat to neural network-…
Adversarial Robustness