An Empirical Analysis of the Advantages of Finite- v.s. Infinite-Width Bayesian Neural Networks
Comparing Bayesian neural networks (BNNs) with different widths is challenging because, as the width increases, multiple model properties change simultaneously, and, inference in the finite-width case is intractable. In this work, we empirically compare finite- and infinite-width BNNs, and provide quantitative and qualitative explanations for their performance difference. We find that when the model is mis-specified, increasing width can hurt BNN performance. In these cases, we provide evidence that finite-width BNNs generalize better partially due to the properties of their frequency spectrum that allows them to adapt under model mismatch.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network
The recent theoretical analysis of deep neural networks in their infinite-width limits has deepened our understanding of initialisation, feature learning, and training of those networks, and brought new practical techniq…
Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
Common infinite-width architectures such as Neural Tangent Kernels (NTKs) have historically shown weak performance compared to finite models. This is usually attributed to the absence of feature learning. We show that th…
Mathematical Foundations of Neural Tangents and Infinite-Width Networks
We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture i…
Infinitely Deep Infinite-Width Networks
Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-wi…
Bounding generalization error with input compression: An empirical study with infinite-width networks
Estimating the Generalization Error (GE) of Deep Neural Networks (DNNs) is an important task that often relies on availability of held-out data. The ability to better predict GE based on a single training set may yield o…