Jointly Fine-Tuning “BERT-like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition

2020-08-15 · Interspeech 2020 8 · Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, Suranga Nanayakkara

Multimodal emotion recognition from speech is an important area in affective computing. Fusing multiple data modalities and learning representations with limited amounts of labeled data is a challenging task. In this paper, we explore the use of modality-specific "BERT-like" pretrained Self Supervised Learning (SSL) architectures to represent both speech and text modalities for the task of multimodal speech emotion recognition. By conducting experiments on three publicly available datasets (IEMOCAP, CMU-MOSEI, and CMU-MOSI), we show that jointly fine-tuning "BERT-like" SSL architectures achieve state-of-the-art (SOTA) results. We also evaluate two methods of fusing speech and text modalities and show that a simple fusion mechanism can outperform more complex ones when using SSL models that have similar architectural properties to BERT.

📄 PDF Abstract BibTeX

Code (1)

shamanez/BERT-like-is-All-You-Need pytorch

Tasks

Emotion Recognition Multimodal Deep Learning Multimodal Emotion Recognition Multimodal Sentiment Analysis Self-Supervised Learning Sentiment Analysis Speech Emotion Recognition Transfer Learning

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Weight Decay 설명 없음

Adam 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

WordPiece 설명 없음

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

2020-08-15 · Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, Suranga Nanayakkara

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

2020-02-24 · Yige Xu, Xipeng Qiu, Ligao Zhou, Xuanjing Huang

Fine-tuning pre-trained language models like BERT has become an effective way in NLP and yields state-of-the-art results on many downstream tasks. Recent studies on adapting BERT to new tasks mainly focus on modifying th…

Natural Language Inference text-classification Text Classification

Improving BERT Fine-tuning with Embedding Normalization

2019-11-10 · Wenxuan Zhou, Junyi Du, Xiang Ren

Large pre-trained sentence encoders like BERT start a new chapter in natural language processing. A common practice to apply pre-trained BERT to sequence classification tasks (e.g., classification of sentences or sentenc…

Classification General Classification Sentence text-classification+1

Joint Encoder-Decoder Self-Supervised Pre-training for ASR

2022-06-09 · Arunkumar A, Umesh S

Self-supervised learning (SSL) has shown tremendous success in various speech-related downstream tasks, including Automatic Speech Recognition (ASR). The output embeddings of the SSL model are treated as powerful short-t…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Language Modelling+3

Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

2019-08-30 · LREC 2020 5 · Alexander Rietzler, Sebastian Stabinger, Paul Opitz, Stefan Engl

Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce, where data and insights from reviews can be leveraged to create value …

Aspect-Based Sentiment Analysis Aspect-Based Sentiment Analysis (ABSA)Domain Adaptation General Classification+5