Stochastic Thermodynamics of Learning Parametric Probabilistic Models
We have formulated a family of machine learning problems as the time evolution of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics: Memorized-information (M-info) and Learned-information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Information thermodynamics: from physics to neuroscience
This paper provides a perspective on applying the concepts of information thermodynamics, developed recently in non-equilibrium statistical physics, to problems in theoretical neuroscience. Historically, information and …
Speed-accuracy relations for diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport
We discuss a connection between a generative model, called the diffusion model, and nonequilibrium thermodynamics for the Fokker-Planck equation, called stochastic thermodynamics. Using techniques from stochastic thermod…
Stochastic Variational Inference
We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilisti…
ArticlesTopic ModelsVariational InferenceLazyPPL: laziness and types in non-parametric probabilistic programs
We introduce LazyPPL, a prototype probabilistic programming library for Haskell. The library emphasises the clarifying power of types, and the connection between non-parametric, stochastic processes and lazy (call by nee…
ClusteringGaussian ProcessesPoint ProcessesProbabilistic ProgrammingStochastic Online Convex Optimization. Application to probabilistic time series forecasting
We introduce a general framework of stochastic online convex optimization to obtain fast-rate stochastic regret bounds. We prove that algorithms such as online newton steps and a scale-free 10 version of Bernstein online…
Probabilistic Time Series ForecastingTime SeriesTime Series AnalysisTime Series Forecasting+1