On the importance of multiple training seeds for evaluating machine unlearning
Machine unlearning aims to remove the influence of certain data points from a trained model without costly retraining. Most practical unlearning algorithms are only approximate and their performance can only be assessed empirically. Common practice is to run unlearning algorithms multiple times independently (i.e., using multiple unlearning seeds) starting from the same trained model (i.e., using only a single training seed ). In image-classification experiments, this practice can give non-representative results as unlearning performance can be sensitive to the choice of training seed. This is particularly relevant for deterministic unlearning methods which always produce the same result when started from the same trained model. Further experiments on federated learning-to-rank, and large language models confirm that this issue extends beyond image classification. We also explain why increasing the number of unlearning seeds cannot generally compensate for the lack of multiple training seeds. Finally, we give guidance on how to select the number of training and unlearning seeds.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationFederated LearningSimilar Papers 제목 키워드 기반
SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation
Machine unlearning removes the influence of specific training data from a trained model without retraining it from scratch. Evaluating an unlearning method requires repeating training, unlearning, and evaluation across m…
Image ClassificationFace RecognitionA novel method for identifying rice seed purity based on hybrid machine learning algorithms
In the grain industry, the identification of seed purity is a crucial task as it is an important factor in evaluating the quality of seeds. For rice seeds, this property allows for the reduction of unexpected influences …
Hybrid Machine LearningParameter Averaging for Feature Ranking
Neural Networks are known to be sensitive to initialisation. The methods that rely on neural networks for feature ranking are not robust since they can have variations in their ranking when the model is initialized and t…
Decision MakingFeature Importancefeature selectionBad Seeds: Evaluating Lexical Methods for Bias Measurement
A common factor in bias measurement methods is the use of hand-curated seed lexicons, but there remains little guidance for their selection. We gather seeds used in prior work, documenting their common sources and ration…
Ablation Study of How Run Time Assurance Impacts the Training and Performance of Reinforcement Learning Agents
Reinforcement Learning (RL) has become an increasingly important research area as the success of machine learning algorithms and methods grows. To combat the safety concerns surrounding the freedom given to RL agents whi…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Exploration+1