MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition
As in school, one teacher to cover all subjects is insufficient to distill equally robust information to a student. Hence, each subject is taught by a highly specialised teacher. Following a similar philosophy, we propose a multiple specialized teacher framework to distill knowledge to a student network. In our approach, directed at face recognition use cases, we train four teachers on one specific ethnicity, leading to four highly specialized and biased teachers. Our strategy learns a project of these four teachers into a common space and distill that information to a student network. Our results highlighted increased performance and reduced bias for all our experiments. In addition, we further show that having biased/specialized teachers is crucial by showing that our approach achieves better results than when knowledge is distilled from four teachers trained on balanced datasets. Our approach represents a step forward to the understanding of the importance of ethnicity-specific features.
Code (1)
Tasks
Face RecognitionKnowledge DistillationPhilosophySimilar Papers 제목 키워드 기반
Wisdom of Committee: Distilling from Foundation Model to Specialized Application Model
Recent advancements in foundation models have yielded impressive performance across a wide range of tasks. Meanwhile, for specific applications, practitioners have been developing specialized application models. To enjoy…
Knowledge DistillationmodelTransfer LearningAdaptive Multi-Teacher Knowledge Distillation with Meta-Learning
Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obta…
Knowledge DistillationMeta-LearningLanguage-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR
Modern LLM-based ASR systems have established multilingual capability as a standard feature, leveraging large-scale multilingual corpora and LLMs' cross-lingual knowledge to achieve competitive performance across multili…
Reinforcement LearningMST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation
Knowledge distillation as an efficient knowledge transfer technique, has achieved remarkable success in unimodal scenarios. However, in cross-modal settings, conventional distillation methods encounter significant challe…
Knowledge DistillationBuilding a Multi-domain Neural Machine Translation Model using Knowledge Distillation
Lack of specialized data makes building a multi-domain neural machine translation tool challenging. Although emerging literature dealing with low resource languages starts to show promising results, most state-of-the-art…
Domain AdaptationKnowledge DistillationMachine TranslationTranslation