Coupled Ensembles of Neural Networks
We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters without losing performance or to significantly improve the performance with the same level of performance. The use of branches brings an additional form of regularization. In addition to the split into parallel branches, we propose a tighter coupling of these branches by placing the "fuse (averaging) layer" before the Log-Likelihood and SoftMax layers during training. This gives another significant performance improvement, the tighter coupling favouring the learning of better representations, even at the level of the individual branches. We refer to this branched architecture as "coupled ensembles". The approach is very generic and can be applied with almost any DCNN architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks. For the same budget, DenseNet-BC has error rate of 3.46%, 17.18%, and 1.8% respectively. With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hopf-type neurons increase input-sensitivity by forming forcing-coupled ensembles
Astounding properties of biological sensors can often be mapped onto a dynamical system in the vicinity a bifurcation. For mammalian hearing, a Hopf bifurcation description has been shown to work across a whole range of …
SensitivityVocal Bursts Type PredictionLow Dimensional Dynamics of Globally Coupled Complex Riccati Equations: Exact Firing-rate Equations for Spiking Neurons with Clustered Substructure
We report on an exact theory for ensembles of globally coupled, heterogeneous complex Riccati equations. A drastic dimensionality reduction to a few ordinary differential equations is achieved for Lorentzian heterogeneit…
Dimensionality ReductionReinforcement learning for suppression of collective activity in oscillatory ensembles
We present a use of modern data-based machine learning approaches to suppress self-sustained collective oscillations typically signaled by ensembles of degenerative neurons in the brain. The proposed hybrid model relies …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes
Training nonlinear parametrizations such as deep neural networks to numerically approximate solutions of partial differential equations is often based on minimizing a loss that includes the residual, which is analyticall…
Diverse Projection Ensembles for Distributional Reinforcement Learning
In contrast to classical reinforcement learning (RL), distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unkno…
Distributional Reinforcement LearningDiversityInductive Biasreinforcement-learning+2