Bayesian interpretation of SGD as Ito process

2019-11-20 · Soma Yokoi, Issei Sato

The current interpretation of stochastic gradient descent (SGD) as a stochastic process lacks generality in that its numerical scheme restricts continuous-time dynamics as well as the loss function and the distribution of gradient noise. We introduce a simplified scheme with milder conditions that flexibly interprets SGD as a discrete-time approximation of an Ito process. The scheme also works as a common foundation of SGD and stochastic gradient Langevin dynamics (SGLD), providing insights into their asymptotic properties. We investigate the convergence of SGD with biased gradient in terms of the equilibrium mode and the overestimation problem of the second moment of SGLD.

📄 PDF Abstract BibTeX arXiv:1911.09011

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Non-trivial informational closure of a Bayesian hyperparameter

2020-10-05 · Martin Biehl, Ryota Kanai

We investigate the non-trivial informational closure (NTIC) of a Bayesian hyperparameter inferring the underlying distribution of an identically and independently distributed finite random variable. For this we embed bot…

Comment on "Modeling rapid language learning by distilling Bayesian priors into artificial neural networks"

2026-08-13 · Orr Well, Idan Tarshish, Nur Lan, Roni Katzir arxiv

McCoy & Griffiths (2025, henceforth M&G) suggest that a Bayesian prior can be distilled into Artificial Neural Networks (ANNs) through Model-Agnostic Meta-Learning (MAML, Finn et al., 2017). They support this empirically…

An Interpretable Probabilistic Approach for Demystifying Black-box Predictive Models

2020-07-21 · Catarina Moreira, Yu-Liang Chou, Mythreyi Velmurugan, Chun Ouyang 외

The use of sophisticated machine learning models for critical decision making is faced with a challenge that these models are often applied as a "black-box". This has led to an increased interest in interpretable machine…

BIG-bench Machine Learning Decision Making Interpretable Machine Learning Prediction

Graded strength of comparative illusions is explained by Bayesian inference

2025-11-18 · Yuhan Zhang, Erxiao Wang, Cory Shain arxiv

Like visual processing, language processing is susceptible to illusions in which people systematically misperceive stimuli. In one such case--the comparative illusion (CI), e.g., More students have been to Russia than I …

Bayesian Inference

Non-Bayesian Post-Model-Selection Estimation as Estimation Under Model Misspecification

2023-08-22 · Nadav Harel, Tirza Routtenberg

In many parameter estimation problems, the exact model is unknown and is assumed to belong to a set of candidate models. In such cases, a predetermined data-based selection rule selects a parametric model from a set of c…

channel selection model Model Selection parameter estimation