Multimodal Data Fusion based on the Global Workspace Theory
We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspace Theory from the field of cognitive science. The GWN achieved average F1 score of 0.92 for discrimination between pain patients and healthy participants and average F1 score = 0.75 for further classification of three pain levels for a patient, both based on the multimodal EmoPain dataset captured from people with chronic pain and healthy people performing different types of exercise movements in unconstrained settings. In these tasks, the GWN significantly outperforms the typical fusion approach of merging by concatenation. We further provide extensive analysis of the behaviour of the GWN and its ability to address uncertainties (hidden noise) in multimodal data.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
An Attention Mechanism for Robust Multimodal Integration in a Global Workspace Architecture
Robust multimodal systems must remain effective when some modalities are noisy, degraded, or unreliable. Existing multimodal fusion methods often learn modality selection jointly with representation learning, making it d…
Representation LearningDeep Learning and the Global Workspace Theory
Recent advances in deep learning have allowed Artificial Intelligence (AI) to reach near human-level performance in many sensory, perceptual, linguistic or cognitive tasks. There is a growing need, however, for novel, br…
Deep LearningTranslationZero-shot cross-modal transfer of Reinforcement Learning policies through a Global Workspace
Humans perceive the world through multiple senses, enabling them to create a comprehensive representation of their surroundings and to generalize information across domains. For instance, when a textual description of a …
AttributeContrastive LearningReinforcement Learning (RL)Semi-supervised Multimodal Representation Learning through a Global Workspace
Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations, or to translate signals from one domain to another (as in imag…
Image CaptioningImage GenerationRepresentation LearningText to Image Generation+2How sustainable are different levels of consciousness?
The human brain processes a wide variety of inputs and does so either consciously or subconsciously. According to the Global Workspace theory, conscious processing involves broadcasting of information to several regions …