On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a specialized variant of MoE that excels in handling complex inputs and improving performance on targeted tasks. Our investigation highlights the advantages of using varied gating functions, moving beyond softmax gating within HMoE frameworks. We theoretically demonstrate that applying tailored gating functions to each expert group allows HMoE to achieve robust results, even when optimal gating functions are applied only at select hierarchical levels. Empirical validation across diverse scenarios supports these theoretical claims. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationMixture-of-ExpertsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression
We propose a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression. Our mixture-of-experts model is conceptually simple and hierarchically recombines computations for an overall…
Mixture-of-ExpertsregressionDropout Regularization in Hierarchical Mixture of Experts
Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defi…
Mixture-of-ExpertsOn Bayesian Softmax-Gated Mixture-of-Experts Models
Mixture-of-experts models provide a flexible framework for learning complex probabilistic input-output relationships by combining multiple expert models through an input-dependent gating mechanism. These models have beco…
Density EstimationConvergence Rates for Softmax Gating Mixture of Experts
Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among multiple specialized sub-models termed e…
Mixture-of-Expertsparameter estimationMoME: Estimating Psychological Traits from Gait with Multi-Stage Mixture of Movement Experts
Gait encodes rich biometric and behavioural information, yet leveraging the manner of walking to infer psychological traits remains a challenging and underexplored problem. We introduce a hierarchical Multi-Stage Mixture…
Gender Prediction