Quantifying Inherent Randomness in Machine Learning Algorithms
Most machine learning (ML) algorithms have several stochastic elements, and their performances are affected by these sources of randomness. This paper uses an empirical study to systematically examine the effects of two sources: randomness in model training and randomness in the partitioning of a dataset into training and test subsets. We quantify and compare the magnitude of the variation in predictive performance for the following ML algorithms: Random Forests (RFs), Gradient Boosting Machines (GBMs), and Feedforward Neural Networks (FFNNs). Among the different algorithms, randomness in model training causes larger variation for FFNNs compared to tree-based methods. This is to be expected as FFNNs have more stochastic elements that are part of their model initialization and training. We also found that random splitting of datasets leads to higher variation compared to the inherent randomness from model training. The variation from data splitting can be a major issue if the original dataset has considerable heterogeneity. Keywords: Model Training, Reproducibility, Variation
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Variational measurement-based quantum computation for generative modeling
Measurement-based quantum computation (MBQC) offers a fundamentally unique paradigm to design quantum algorithms. Indeed, due to the inherent randomness of quantum measurements, the natural operations in MBQC are not det…
Machine Learning vs. Randomness: Challenges in Predicting Binary Options Movements
Binary options trading is often marketed as a field where predictive models can generate consistent profits. However, the inherent randomness and stochastic nature of binary options make price movements highly unpredicta…
Hyperparameter OptimizationBetween Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale
To develop rigorous knowledge about ML models -- and the systems in which they are embedded -- we need reliable measurements. But reliable measurement is fundamentally challenging, and touches on issues of reproducibilit…
MemorizationUncertainty QuantificationThe Normal Distributions Indistinguishability Spectrum and its Application to Privacy-Preserving Machine Learning
Differential Privacy (DP) (and its variants) is the most common method for machine learning (ML) on privacy-sensitive data. In big data analytics, one often uses randomized sketching/aggregation algorithms to make proces…
Privacy PreservingComparing hundreds of machine learning classifiers and discrete choice models in predicting travel behavior: an empirical benchmark
Numerous studies have compared machine learning (ML) and discrete choice models (DCMs) in predicting travel demand. However, these studies often lack generalizability as they compare models deterministically without cons…
Computational EfficiencyDiscrete Choice ModelsExperimental Design