Overcoming the Domain Gap in Neural Action Representations
Relating animal behaviors to brain activity is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations exploiting the properties of microscopy imaging. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions. To demonstrate this, we test our methods on three very different multimodal datasets; one that features flies and their neural activity, one that contains human neural Electrocorticography (ECoG) data, and lastly the RGB video data of human activities from different viewpoints.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic
Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust…
Open Vocabulary Action RecognitionZero-shot GeneralizationOvercoming the Domain Gap in Contrastive Learning of Neural Action Representations
A fundamental goal in neuroscience is to understand the relationship between neural activity and behavior. For example, the ability to extract behavioral intentions from neural data, or neural decoding, is critical for d…
Contrastive LearningDomain AdaptationMarkerless Motion CaptureSelf-Supervised Learning+1Domain-Aware Dialogue State Tracker for Multi-Domain Dialogue Systems
In task-oriented dialogue systems the dialogue state tracker (DST) component is responsible for predicting the state of the dialogue based on the dialogue history. Current DST approaches rely on a predefined domain ontol…
Language ModelingLanguage ModellingTask-Oriented Dialogue Systemsplayer2vec: A Language Modeling Approach to Understand Player Behavior in Games
Methods for learning latent user representations from historical behavior logs have gained traction for recommendation tasks in e-commerce, content streaming, and other settings. However, this area still remains relative…
Language ModelingLanguage ModellingMISO: Mutual Information Loss with Stochastic Style Representations for Multimodal Image-to-Image Translation
Unpaired multimodal image-to-image translation is a task of translating a given image in a source domain into diverse images in the target domain, overcoming the limitation of one-to-one mapping. Existing multimodal tran…
Image ReconstructionImage-to-Image TranslationTranslation