On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm
In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellstr\"om and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and we refine them for loss functions with potentially large gradient norms.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Generalization Bounds via Information Density and Conditional Information Density
We present a general approach, based on exponential inequalities, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error…
Generalization BoundsGeneralization Error Bounds for Noisy, Iterative Algorithms
In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bo…
Learning TheoryGeneralization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data p…
Generalization BoundsInformation-Theoretic Generalization Bounds for Stochastic Gradient Descent
We study the generalization properties of the popular stochastic optimization method known as stochastic gradient descent (SGD) for optimizing general non-convex loss functions. Our main contribution is providing upper b…
Generalization BoundsStochastic OptimizationStability, Complexity and Data-Dependent Worst-Case Generalization Bounds
Providing generalization guarantees for stochastic optimization algorithms remains a key challenge in learning theory. Recently, numerous works demonstrated the impact of the geometric properties of optimization trajecto…
Stochastic Optimization