Approximating Probability Distributions by ReLU Networks
How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Learning Distributions by Generative Adversarial Networks: Approximation and Generalization
We study how well generative adversarial networks (GAN) learn probability distributions from finite samples by analyzing the convergence rates of these models. Our analysis is based on a new oracle inequality that decomp…
Generalization BoundsLearning TheoryUnderstanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks
Motivated by challenges in conditional generative modeling, where the target conditional density takes the form of a ratio f1 over f2, this paper develops a theoretical framework for approximating such ratio-type functio…
Agnostic Learning of General ReLU Activation Using Gradient Descent
We provide a convergence analysis of gradient descent for the problem of agnostically learning a single ReLU function with moderate bias under Gaussian distributions. Unlike prior work that studies the setting of zero bi…
ReLU Networks as Random Functions: Their Distribution in Probability Space
This paper presents a novel framework for understanding trained ReLU networks as random, affine functions, where the randomness is induced by the distribution over the inputs. By characterizing the probability distributi…
Dying ReLU and Initialization: Theory and Numerical Examples
The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theo…