Quasi-Newton Quasi-Monte Carlo for variational Bayes

2021-04-07 · Sifan Liu, Art B. Owen

Many machine learning problems optimize an objective that must be measured with noise. The primary method is a first order stochastic gradient descent using one or more Monte Carlo (MC) samples at each step. There are settings where ill-conditioning makes second order methods such as L-BFGS more effective. We study the use of randomized quasi-Monte Carlo (RQMC) sampling for such problems. When MC sampling has a root mean squared error (RMSE) of $O(n^{-1/2})$ then RQMC has an RMSE of $o(n^{-1/2})$ that can be close to $O(n^{-3/2})$ in favorable settings. We prove that improved sampling accuracy translates directly to improved optimization. In our empirical investigations for variational Bayes, using RQMC with stochastic L-BFGS greatly speeds up the optimization, and sometimes finds a better parameter value than MC does.

📄 PDF Abstract BibTeX arXiv:2104.02865

Code (0)

등록된 구현이 없습니다.

Tasks

Second-order methods

Quasi-Newton Methods for Markov Chain Monte Carlo

2011-12-01 · NeurIPS 2011 12 · Yichuan Zhang, Charles A. Sutton

The performance of Markov chain Monte Carlo methods is often sensitive to the scaling and correlations between the random variables of interest. An important source of information about the local correlation and scale is…

valid

Pathfinder: Parallel quasi-Newton variational inference

2021-08-09 · Lu Zhang, Bob Carpenter, Andrew Gelman, Aki Vehtari

We propose Pathfinder, a variational method for approximately sampling from differentiable log densities. Starting from a random initialization, Pathfinder locates normal approximations to the target density along a quas…

Pathfinder Variational Inference

Quasi-Monte Carlo Variational Inference

2018-07-04 · ICML 2018 7 · Alexander Buchholz, Florian Wenzel, Stephan Mandt

Many machine learning problems involve Monte Carlo gradient estimators. As a prominent example, we focus on Monte Carlo variational inference (MCVI) in this paper. The performance of MCVI crucially depends on the varianc…

Variational Inference

Stochastic Quasi-Newton Langevin Monte Carlo

2016-02-10 · Umut Şimşekli, Roland Badeau, A. Taylan Cemgil, Gaël Richard

Recently, Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods have been proposed for scaling up Monte Carlo computations to large data problems. Whilst these approaches have proven useful in many applications,…

Second-order methods

Quasi-Monte Carlo sampling for machine-learning partial differential equations

2019-11-05 · Jingrun Chen, Rui Du, Panchi Li, Liyao Lyu

Solving partial differential equations in high dimensions by deep neural network has brought significant attentions in recent years. In many scenarios, the loss function is defined as an integral over a high-dimensional …

BIG-bench Machine Learning Numerical Integration