Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams
Video game streaming provides the viewer with a rich set of audio-visual data, conveying information both with regards to the game itself, through game footage and audio, as well as the streamer's emotional state and behaviour via webcam footage and audio. Analysing player behaviour and discovering correlations with game context is crucial for modelling and understanding important aspects of livestreams, but comes with a significant set of challenges - such as fusing multimodal data captured by different sensors in uncontrolled ('in-the-wild') conditions. Firstly, we present, to our knowledge, the first data set of League of Legends livestreams, annotated for both streamer affect and game context. Secondly, we propose a method that exploits tensor decompositions for high-order fusion of multimodal representations. The proposed method is evaluated on the problem of jointly predicting game context and player affect, compared with a set of baseline fusion approaches such as late and early fusion.
Code (1)
Similar Papers 제목 키워드 기반
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition
Multimodal emotion recognition aims to recognize emotions for each utterance of multiple modalities, which has received increasing attention for its application in human-machine interaction. Current graph-based methods f…
Contrastive LearningEmotion RecognitionEmotion Recognition in ConversationFace Swapping+1CG-MER: A Card Game-based Multimodal dataset for Emotion Recognition
The field of affective computing has seen significant advancements in exploring the relationship between emotions and emerging technologies. This paper presents a novel and valuable contribution to this field with the in…
Emotion RecognitionNavigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition
Multimodal emotion recognition aims to integrate text, audio, and video sources to understand human affective states. Although multimodal large language models excel at multimodal reasoning, they typically treat emotion …
Multimodal Emotion RecognitionEmotion ClassificationMultimodal ReasoningFuzzy Approach for Audio-Video Emotion Recognition in Computer Games for Children
Computer games are widespread nowadays and enjoyed by people of all ages. But when it comes to kids, playing these games can be more than just fun, it is a way for them to develop important skills and build emotional int…
Audio Emotion RecognitionEmotional IntelligenceEmotion RecognitionVideo Emotion RecognitionCross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition
Emotion recognition has a wide range of applications in human-computer interaction, marketing, healthcare, and other fields. In recent years, the development of deep learning technology has provided new methods for emoti…
cross-modal alignmentEmotion ClassificationEmotion RecognitionMarketing+1