On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Means
In this paper, we study a simple algorithm to construct asymptotically valid confidence regions for model parameters using the batch means method. The main idea is to cancel out the covariance matrix which is hard/costly to estimate. In the process of developing the algorithm, we establish process-level functional central limit theorem for Polyak-Ruppert averaging based stochastic gradient descent estimators. We also extend the batch means method to accommodate more general batch size specifications.
Code (0)
등록된 구현이 없습니다.
Tasks
validSimilar Papers 제목 키워드 기반
Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance
Stochastic gradient descent (SGD) is a foundational algorithm for large-scale statistical learning and stochastic optimization. However, statistical inference based on SGD iterates remains challenging when stochastic gra…
Stochastic OptimizationInformation-Theoretic Trust Regions for Stochastic Gradient-Based Optimization
Stochastic gradient-based optimization is crucial to optimize neural networks. While popular approaches heuristically adapt the step size and direction by rescaling gradients, a more principled approach to improve optimi…
Finite Sample Confidence Regions for Linear Regression Parameters Using Arbitrary Predictors
We explore a novel methodology for constructing confidence regions for parameters of linear models, using predictions from any arbitrary predictor. Our framework requires minimal assumptions on the noise and can be exten…
regressionOn The Verification of Neural ODEs with Stochastic Guarantees
We show that Neural ODEs, an emerging class of time-continuous neural networks, can be verified by solving a set of global-optimization problems. For this purpose, we introduce Stochastic Lagrangian Reachability (SLR), a…
global-optimizationStatistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm
Statistical machine learning models trained with stochastic gradient algorithms are increasingly being deployed in critical scientific applications. However, computing the stochastic gradient in several such applications…
BIG-bench Machine Learningparameter estimationvalid