Cross-Modal BERT for Text-Audio Sentiment Analysis
Multimodal sentiment analysis is an emerging research field that aims to enable machines to recognize, interpret, and express emotion. Through the cross-modal interaction, we can get more comprehensive emotional characteristics of the speaker. Bidirectional Encoder Representations from Transformers (BERT) is an efficient pre-trained language representation model. Fine-tuning it has obtained new state-of-the-art results on eleven natural language processing tasks like question answering and natural language inference. However, most previous works fine-tune BERT only base on text data, how to learn a better representation by introducing the multimodal information is still worth exploring. In this paper, we propose the Cross-Modal BERT (CM-BERT), which relies on the interaction of text and audio modality to fine-tune the pre-trained BERT model. As the core unit of the CM-BERT, masked multimodal attention is designed to dynamically adjust the weight of words by combining the information of text and audio modality. We evaluate our method on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that it has significantly improved the performance on all the metrics over previous baselines and text-only finetuning of BERT. Besides, we visualize the masked multimodal attention and proves that it can reasonably adjust the weight of words by introducing audio modality information.
Code (1)
Tasks
Multimodal Sentiment AnalysisNatural Language InferenceQuestion AnsweringSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis
Multimodal learning pipelines have benefited from the success of pretrained language models. However, this comes at the cost of increased model parameters. In this work, we propose Adapted Multimodal BERT (AMB), a BERT-b…
Language ModelingLanguage ModellingSentiment AnalysisMultimodal Sentiment Analysis on CMU-MOSEI Dataset using Transformer-based Models
This project performs multimodal sentiment analysis using the CMU-MOSEI dataset, using transformer-based models with early fusion to integrate text, audio, and visual modalities. We employ BERT-based encoders for each mo…
Multimodal Sentiment AnalysisSentiment AnalysisAudio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio f…
Machine TranslationSpeech RecognitionSentiment AnalysisEnriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
Multimodal Sentiment Analysis (MSA) stands as a critical research frontier, seeking to comprehensively unravel human emotions by amalgamating text, audio, and visual data. Yet, discerning subtle emotional nuances within …
Multimodal Sentiment AnalysisSentiment AnalysisMultimodal Sentiment Analysis Based on BERT and ResNet
With the rapid development of the Internet and social media, multi-modal data (text and image) is increasingly important in sentiment analysis tasks. However, the existing methods are difficult to effectively fuse text a…
Multimodal Sentiment AnalysisSentiment Analysis