Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis
Modality representation learning is an important problem for multimodal sentiment analysis (MSA), since the highly distinguishable representations can contribute to improving the analysis effect. Previous works of MSA have usually focused on multimodal fusion strategies, and the deep study of modal representation learning was given less attention. Recently, contrastive learning has been confirmed effective at endowing the learned representation with stronger discriminate ability. Inspired by this, we explore the improvement approaches of modality representation with contrastive learning in this study. To this end, we devise a three-stages framework with multi-view contrastive learning to refine representations for the specific objectives. At the first stage, for the improvement of unimodal representations, we employ the supervised contrastive learning to pull samples within the same class together while the other samples are pushed apart. At the second stage, a self-supervised contrastive learning is designed for the improvement of the distilled unimodal representations after cross-modal interaction. At last, we leverage again the supervised contrastive learning to enhance the fused multimodal representation. After all the contrast trainings, we next achieve the classification task based on frozen representations. We conduct experiments on three open datasets, and results show the advance of our model.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningMultimodal Sentiment AnalysisRepresentation LearningSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition
Multimodal emotion recognition aims to recognize emotions for each utterance of multiple modalities, which has received increasing attention for its application in human-machine interaction. Current graph-based methods f…
Contrastive LearningEmotion RecognitionEmotion Recognition in ConversationFace Swapping+1Identifiability Results for Multimodal Contrastive Learning
Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a li…
Contrastive LearningRepresentation LearningMultimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis
Multimodal representation learning is a challenging task in which previous work mostly focus on either uni-modality pre-training or cross-modality fusion. In fact, we regard modeling multimodal representation as building…
Contrastive LearningMultimodal Sentiment AnalysisRepresentation LearningSentiment AnalysisUniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning
Multimodal learning aims to imitate human beings to acquire complementary information from multiple modalities for various downstream tasks. However, traditional aggregation-based multimodal fusion methods ignore the int…
Contrastive LearningImage-text Classificationtext-classificationText ClassificationFactorized Contrastive Learning: Going Beyond Multi-view Redundancy
In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., ima…
Contrastive LearningRepresentation Learning