Self-Supervised learning with cross-modal transformers for emotion recognition
Emotion recognition is a challenging task due to limited availability of in-the-wild labeled datasets. Self-supervised learning has shown improvements on tasks with limited labeled datasets in domains like speech and natural language. Models such as BERT learn to incorporate context in word embeddings, which translates to improved performance in downstream tasks like question answering. In this work, we extend self-supervised training to multi-modal applications. We learn multi-modal representations using a transformer trained on the masked language modeling task with audio, visual and text features. This model is fine-tuned on the downstream task of emotion recognition. Our results on the CMU-MOSEI dataset show that this pre-training technique can improve the emotion recognition performance by up to 3% compared to the baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
Emotion RecognitionLanguage ModelingLanguage ModellingMasked Language ModelingQuestion AnsweringSelf-Supervised LearningWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature Fusion
Emotion Recognition is a challenging research area given its complex nature, and humans express emotional cues across various modalities such as language, facial expressions, and speech. Representation and fusion of feat…
Emotion RecognitionMultimodal Deep LearningMultimodal Emotion RecognitionMultimodal Sentiment Analysis+2Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers
Due to the complex nature of human emotions and the diversity of emotion representation methods in humans, emotion recognition is a challenging field. In this research, three input modalities, namely text, audio (speech)…
DiversityEmotion RecognitionMultimodal Emotion RecognitionSelf-Supervised Learning+1Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between au…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion RecognitionFace Reconstruction+5HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition
Emotion recognition in conversations is challenging due to the multi-modal nature of the emotion expression. We propose a hierarchical cross-attention model (HCAM) approach to multi-modal emotion recognition using a comb…
Emotion ClassificationEmotion RecognitionEmotion Recognition in ConversationMultimodal Emotion RecognitionTransformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition
Recently, wearable emotion recognition based on peripheral physiological signals has drawn massive attention due to its less invasive nature and its applicability in real-life scenarios. However, how to effectively fuse …
Emotion ClassificationEmotion RecognitionRepresentation LearningSelf-Supervised Learning