A Latent Variational Framework for Stochastic Optimization
This paper provides a unifying theoretical framework for stochastic optimization algorithms by means of a latent stochastic variational problem. Using techniques from stochastic control, the solution to the variational problem is shown to be equivalent to that of a Forward Backward Stochastic Differential Equation (FBSDE). By solving these equations, we recover a variety of existing adaptive stochastic gradient descent methods. This framework establishes a direct connection between stochastic optimization algorithms and a secondary Bayesian inference problem on gradients, where a prior measure on noisy gradient observations determines the resulting algorithm.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceStochastic OptimizationSimilar Papers 제목 키워드 기반
Stochastic Deep Learning: A Probabilistic Framework for Modeling Uncertainty in Structured Temporal Data
I propose a novel framework that integrates stochastic differential equations (SDEs) with deep generative models to improve uncertainty quantification in machine learning applications involving structured and temporal da…
Smoothed Gradients for Stochastic Variational Inference
Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with …
Stochastic OptimizationVariational InferenceMISSO: Minimization by Incremental Stochastic Surrogate Optimization for Large Scale Nonconvex and Nonsmooth Problems
Many constrained, nonconvex and nonsmooth optimization problems can be tackled using the majorization-minimization (MM) method which alternates between constructing a surrogate function which upper bounds the objective f…
Variational InferenceRobust, Accurate Stochastic Optimization for Variational Inference
We consider the problem of fitting variational posterior approximations using stochastic optimization methods. The performance of these approximations depends on (1) how well the variational family matches the true poste…
DiagnosticStochastic OptimizationVariational InferenceStochastic Collapsed Variational Inference for Structured Gaussian Process Regression Network
This paper presents an efficient variational inference framework for deriving a family of structured gaussian process regression network (SGPRN) models. The key idea is to incorporate auxiliary inducing variables in late…
ImputationMissing ValuesregressionStochastic Optimization+1