On Nondeterminism and Instability in Optimizing Neural Networks
Nondeterminism in optimization causes uncertainty when improving neural networks, with small gains in performance difficult to discern from run-to-run variability. While this uncertainty can be reduced by training multiple copies of a model with different random seeds, doing so can be time-consuming, costly, and is energy-inefficient. Despite this, little attention has been paid towards establishing an understanding of this problem. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, which allows us to study the independent effects of nondeterminism sources, including random parameter initialization, data augmentation, data shuffling, and even low-level nondeterminism in popular accelerator libraries like cuDNN. Surprisingly, we find that changes to each source of nondeterminism all have similar effects on measures of model diversity. To explain this intriguing fact, we examine and identify the instability of model training, when taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial model parameters result in models that converge to vastly different values, and even place a lower bound of $10^7$ on the condition number of model training for example architectures considered.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationDiversitySimilar Papers 제목 키워드 기반
Nondeterminism and Instability in Neural Network Optimization
Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model c…
DiversityOn Nondeterminism and Instability in Neural Network Optimization
Optimization nondeterminism causes uncertainty when improving neural networks, with small changes in performance difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple cop…
DiversityDeterministic Implementations for Reproducibility in Deep Reinforcement Learning
While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminis…
Deep Reinforcement LearningQ-Learningreinforcement-learningReinforcement Learning+1Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism
The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results. This is due t…
Curriculum Abductive Learning
Abductive Learning (ABL) integrates machine learning with logical reasoning in a loop: a learning model predicts symbolic concept labels from raw inputs, which are revised through abduction using domain knowledge and the…
Logical Reasoning