Towards a Universal Gating Network for Mixtures of Experts
The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to specific networks based on the nature of the expert activations generated. The experiments revealed that the gating networks, including the universal gating approach, constituted the most accurate approach, and therefore represent a pragmatic step towards applications with heterogeneous mixtures of experts in a data-free regime. The code for this project is hosted on github at https://github.com/cwkang1998/network-merging.
Code (1)
Similar Papers 제목 키워드 기반
Tighter Risk Bounds for Mixtures of Experts
In this work, we provide upper bounds on the risk of mixtures of experts by imposing local differential privacy (LDP) on their gating mechanism. These theoretical guarantees are tailored to mixtures of experts that utili…
Gaussian Process-Gated Hierarchical Mixtures of Experts
In this paper, we propose novel Gaussian process-gated hierarchical mixtures of experts (GPHMEs). Unlike other mixtures of experts with gating models linear in the input, our model employs gating functions built with Gau…
Gaussian ProcessesVariational InferenceBiased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations
We propose a novel mixture-of-experts class to optimize computer vision models in accordance with data transfer limitations at test time. Our approach postulates that the minimum acceptable amount of data allowing for hi…
Action ClassificationImage Super-ResolutionMixture-of-ExpertsSuper-ResolutionHorseshoe Mixtures-of-Experts (HS-MoE)
Horseshoe mixtures-of-experts (HS-MoE) models provide a Bayesian framework for sparse expert selection in mixture-of-experts architectures. We combine the horseshoe prior's adaptive global-local shrinkage with input-depe…
Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation
Vulnerability to adversarial attacks is a well-known deficiency of deep neural networks. Larger networks are generally more robust, and ensembling is one method to increase adversarial robustness: each model's weaknesses…
Adversarial RobustnessMixture-of-ExpertsSemantic Segmentation