ModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation

2024-08-28 · Tiantian Feng, Tuo Zhang, Salman Avestimehr, Shrikanth S. Narayanan

Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning. It is particularly prevalent in audiovisual learning, with audio is often assumed to be the weaker modality in recognition tasks. To address this challenge, we introduce ModalityMirror to improve audio model performance by leveraging knowledge distillation from an audiovisual federated learning model. ModalityMirror involves two phases: a modality-wise FL stage to aggregate uni-modal encoders; and a federated knowledge distillation stage on multi-modality clients to train an unimodal student model. Our results demonstrate that ModalityMirror significantly improves the audio classification compared to the state-of-the-art FL methods such as Harmony, particularly in audiovisual FL facing video missing. Our approach unlocks the potential for exploiting the diverse modality spectrum inherent in multi-modal FL.

📄 PDF Abstract BibTeX arXiv:2408.15803

Code (0)

등록된 구현이 없습니다.

Tasks

Audio Classification Federated Learning Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

3FM: Multi-modal Meta-learning for Federated Tasks

2023-12-15 · Minh Tran, Roochi Shah, Zejun Gong

We present a novel approach in the domain of federated learning (FL), particularly focusing on addressing the challenges posed by modality heterogeneity, variability in modality availability across clients, and the preva…

Federated Learning Meta-Learning

Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning

2024-01-01 · CVPR 2024 1 · Fan Qi, Shuai Li

In Federated Learning (FL) the issue of statistical data heterogeneity has been a significant challenge to the field's ongoing development. This problem is further exacerbated when clients' data vary in modalities. I…

Federated Learning Graph Learning

MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity

2026-08-14 · Adiba Orzikulova, Dong Min Kim, Jaehong Yoon, Sung-Ju Lee arxiv

Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may co…

Federated Learning

Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity

2026-08-01 · Yinlin Zhu, Di Wu, Yi Zhang, Xunkai Li 외 arxiv

Multimodal-attributed graphs (MAGs), where nodes carry heterogeneous semantic content across multiple modalities while edges encode relational dependencies, have been widely adopted across diverse domains. Federated mult…

Contrastive Learning Graph Learning

Mix-modal Federated Learning for MRI Image Segmentation

2025-09-02 · Guyue Hu, Siyuan Song, Jingpeng Sun, Zhe Jin 외 arxiv

Magnetic resonance imaging (MRI) image segmentation is crucial in diagnosing and treating many diseases, such as brain tumors. Existing MRI image segmentation methods mainly fall into a centralized multimodal paradigm, w…

Federated Learning Image Segmentation