HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
Stochastic gradient descent (SGD) is an immensely popular approach for online learning in settings where data arrives in a stream or data sizes are very large. However, despite an ever-increasing volume of work on SGD, much less is known about the statistical inferential properties of SGD-based predictions. Taking a fully inferential viewpoint, this paper introduces a novel procedure termed HiGrad to conduct statistical inference for online learning, without incurring additional computational cost compared with SGD. The HiGrad procedure begins by performing SGD updates for a while and then splits the single thread into several threads, and this procedure hierarchically operates in this fashion along each thread. With predictions provided by multiple threads in place, a $t$-based confidence interval is constructed by decorrelating predictions using covariance structures given by a Donsker-style extension of the Ruppert--Polyak averaging scheme, which is a technical contribution of independent interest. Under certain regularity conditions, the HiGrad confidence interval is shown to attain asymptotically exact coverage probability. Finally, the performance of HiGrad is evaluated through extensive simulation studies and a real data example. An R package \texttt{higrad} has been developed to implement the method.
Code (0)
등록된 구현이 없습니다.
Tasks
Uncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Resampling Stochastic Gradient Descent Cheaply for Efficient Uncertainty Quantification
Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained s…
Stochastic OptimizationUncertainty QuantificationNeuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
Reliable uncertainty quantification in continuous-time (CT) representation learning remains nascent, particularly within CT attention literature. We introduce the Neuronal Stochastic Attention Circuit (NSAC), a novel bio…
Representation LearningAutonomous VehiclesQuantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems
Physics-informed neural networks (PINNs) have recently emerged as an alternative way of solving partial differential equations (PDEs) without the need of building elaborate grids, instead, using a straightforward impleme…
Active LearningUncertainty QuantificationAssessment of DeepONet for reliability analysis of stochastic nonlinear dynamical systems
Time dependent reliability analysis and uncertainty quantification of structural system subjected to stochastic forcing function is a challenging endeavour as it necessitates considerable computational time. We investiga…
Uncertainty QuantificationZero-Shot LearningSurrogate modeling for uncertainty quantification in nonlinear dynamics
Predicting the behavior of complex systems in engineering often involves significant uncertainty about operating conditions, such as external loads, environmental effects, and manufacturing variability. As a result, unce…