Cross-Modal Knowledge Transfer via Inter-Modal Translation and Alignment for Affect Recognition
Multi-modal affect recognition models leverage complementary information in different modalities to outperform their uni-modal counterparts. However, due to the unavailability of modality-specific sensors or data, multi-modal models may not be always employable. For this reason, we aim to improve the performance of uni-modal affect recognition models by transferring knowledge from a better-performing (or stronger) modality to a weaker modality during training. Our proposed multi-modal training framework for cross-modal knowledge transfer relies on two main steps. First, an encoder-classifier model creates task-specific representations for the stronger modality. Then, cross-modal translation generates multi-modal intermediate representations, which are also aligned in the latent space with the stronger modality representations. To exploit the contextual information in temporal sequential affect data, we use Bi-GRU and transformer encoder. We validate our approach on two multi-modal affect datasets, namely CMU-MOSI for binary sentiment classification and RECOLA for dimensional emotion regression. The results show that the proposed approach consistently improves the uni-modal test-time performance of the weaker modalities.
Code (0)
등록된 구현이 없습니다.
Tasks
Sentiment AnalysisSentiment ClassificationTransfer LearningTranslationSimilar Papers 제목 키워드 기반
MHTN: Modal-adversarial Hybrid Transfer Network for Cross-modal Retrieval
Cross-modal retrieval has drawn wide interest for retrieval across different modalities of data. However, existing methods based on DNN face the challenge of insufficient cross-modal training data, which limits the train…
Cross-Modal RetrievalRepresentation LearningRetrievalTransfer LearningCross-modal Common Representation Learning by Hybrid Transfer Network
DNN-based cross-modal retrieval is a research hotspot to retrieve across different modalities as image and text, but existing methods often face the challenge of insufficient cross-modal training data. In single-modal sc…
Cross-Modal RetrievalRepresentation LearningRetrievalLearning Modality Knowledge Alignment for Cross-Modality Transfer
Cross-modality transfer aims to leverage large pretrained models to complete tasks that may not belong to the modality of pretraining data. Existing works achieve certain success in extending classical finetuning to cros…
Meta-LearningA Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition
Despite living in a multi-sensory world, most AI models are limited to textual and visual understanding of human motion and behavior. In fact, full situational awareness of human motion could best be understood through a…
Action RecognitionActivity RecognitionDomain AdaptationHuman Activity Recognition+3C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation
Existing Knowledge Distillation (KD) methods typically focus on transferring knowledge from a large-capacity teacher to a low-capacity student model achieving substantial success in unimodal knowledge transfer. Howev…
Knowledge DistillationTransfer Learning