Predicting distributions with Linearizing Belief Networks
Conditional belief networks introduce stochastic binary variables in neural networks. Contrary to a classical neural network, a belief network can predict more than the expected value of the output $Y$ given the input $X$. It can predict a distribution of outputs $Y$ which is useful when an input can admit multiple outputs whose average is not necessarily a valid answer. Such networks are particularly relevant to inverse problems such as image prediction for denoising, or text to speech. However, traditional sigmoid belief networks are hard to train and are not suited to continuous problems. This work introduces a new family of networks called linearizing belief nets or LBNs. A LBN decomposes into a deep linear network where each linear unit can be turned on or off by non-deterministic binary latent units. It is a universal approximator of real-valued conditional distributions and can be trained using gradient descent. Moreover, the linear pathways efficiently propagate continuous information and they act as multiplicative skip-connections that help optimization by removing gradient diffusion. This yields a model which trains efficiently and improves the state-of-the-art on image denoising and facial expression generation with the Toronto faces dataset.
Code (1)
Tasks
DenoisingFacial expression generationImage Denoisingtext-to-speechText to SpeechvalidSimilar Papers 제목 키워드 기반
GP-SUM. Gaussian Processes Filtering of non-Gaussian Beliefs
This work studies the problem of stochastic dynamic filtering and state propagation with complex beliefs. The main contribution is GP-SUM, a filtering algorithm tailored to dynamic systems and observation models expresse…
AttributeGaussian ProcessesThe Linearization of Belief Propagation on Pairwise Markov Networks
Belief Propagation (BP) is a widely used approximation for exact probabilistic inference in graphical models, such as Markov Random Fields (MRFs). In graphs with cycles, however, no exact convergence guarantees for BP ar…
Node ClassificationFeedback Linearization for Unknown Systems via Reinforcement Learning
We present a novel approach to control design for nonlinear systems which leverages model-free policy optimization techniques to learn a linearizing controller for a physical plant with unknown dynamics. Feedback lineari…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Feasible Conditional Belief Distributions
Agents receive private signals about an unknown state. The resulting joint belief distributions are complex and lack a simple characterization. Our key insight is that, when conditioned on the state, the structure of bel…
LoLCATs: On Low-Rank Linearizing of Large Language Models
Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear attention -- avoiding the expensive pretrai…
MMLU