Large Model for Small Data: Foundation Model for Cross-Modal RF Human Activity Recognition
Radio-Frequency (RF)-based Human Activity Recognition (HAR) rises as a promising solution for applications unamenable to techniques requiring computer visions. However, the scarcity of labeled RF data due to their non-interpretable nature poses a significant obstacle. Thanks to the recent breakthrough of foundation models (FMs), extracting deep semantic insights from unlabeled visual data become viable, yet these vision-based FMs fall short when applied to small RF datasets. To bridge this gap, we introduce FM-Fi, an innovative cross-modal framework engineered to translate the knowledge of vision-based FMs for enhancing RF-based HAR systems. FM-Fi involves a novel cross-modal contrastive knowledge distillation mechanism, enabling an RF encoder to inherit the interpretative power of FMs for achieving zero-shot learning. It also employs the intrinsic capabilities of FM and RF to remove extraneous features for better alignment between the two modalities. The framework is further refined through metric-based few-shot learning techniques, aiming to boost the performance for predefined HAR tasks. Comprehensive evaluations evidently indicate that FM-Fi rivals the effectiveness of vision-based methodologies, and the evaluation results provide empirical validation of FM-Fi's generalizability across various environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Activity RecognitionFew-Shot LearningHuman Activity RecognitionKnowledge DistillationmodelZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
Multimodal clinical prediction faces three challenges: multiple foundation models (FMs) with complementary strengths per modality, pervasive missing modalities at training and test time, and sample-specific variation in …
Medical DiagnosisCLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Recent advances in clinical AI have enabled remarkable progress across many clinical domains. However, existing benchmarks and models are primarily limited to a small set of modalities and tasks, which hinders the develo…
FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales
Foundation models offer a promising route to transferable remote sensing representations, but many current approaches depend on very large pretraining datasets and fixed sensor configurations, limiting their suitability …
Scene ClassificationReverso: Efficient Time Series Foundation Models for Zero-shot Forecasting
Learning time series foundation models has been shown to be a promising approach for zero-shot time series forecasting across diverse time series domains. Insofar as scaling has been a critical driver of performance of f…
Time Series ForecastingData AugmentationBioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs
Foundation models (FMs) are able to leverage large volumes of unlabeled data to demonstrate superior performance across a wide range of tasks. However, FMs developed for biomedical domains have largely remained unimodal,…
Cross-Modal RetrievalDomain GeneralizationKnowledge GraphsQuestion Answering+1