Nondeterminism and Instability in Neural Network Optimization
Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, allowing us to isolate the effects of a variety of sources of nondeterminism. Surprisingly, we find that all sources of nondeterminism have similar effects on measures of model diversity. To explain this intriguing fact, we identify the instability of model training, taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial parameters result in models converging to vastly different values. Last, we propose two approaches for reducing the effects of instability on run-to-run variability.
Code (1)
Tasks
DiversitySimilar Papers 제목 키워드 기반
On Nondeterminism and Instability in Neural Network Optimization
Optimization nondeterminism causes uncertainty when improving neural networks, with small changes in performance difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple cop…
DiversityOn Nondeterminism and Instability in Optimizing Neural Networks
Nondeterminism in optimization causes uncertainty when improving neural networks, with small gains in performance difficult to discern from run-to-run variability. While this uncertainty can be reduced by training multip…
Data AugmentationDiversityDeterministic Implementations for Reproducibility in Deep Reinforcement Learning
While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminis…
Deep Reinforcement LearningQ-Learningreinforcement-learningReinforcement Learning+1Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism
The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results. This is due t…
Curriculum Abductive Learning
Abductive Learning (ABL) integrates machine learning with logical reasoning in a loop: a learning model predicts symbolic concept labels from raw inputs, which are revised through abduction using domain knowledge and the…
Logical Reasoning