paper-with-me

홈 › Papers

Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

2024-07-25 · Anna Bavaresco, Marianne de Heer Kloots, Sandro Pezzelle, Raquel Fernández

Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either visual or textual input but not both. Yet, there is growing evidence that human meaning representations integrate linguistic and sensory-motor information. Here we investigate whether the integration of multimodal information operated by current vision-and-language DNN models (VLMs) leads to representations that are more aligned with human brain activity than those obtained by language-only and vision-only DNNs. We focus on fMRI responses recorded while participants read concept words in the context of either a full sentence or an accompanying picture. Our results reveal that VLM representations correlate more strongly than language- and vision-only DNNs with activations in brain areas functionally related to language processing. A comparison between different types of visuo-linguistic architectures shows that recent generative VLMs tend to be less brain-aligned than previous architectures with lower performance on downstream applications. Moreover, through an additional analysis comparing brain vs. behavioural alignment across multiple VLMs, we show that -- with one remarkable exception -- representations that strongly align with behavioural judgments do not correlate highly with brain responses. This indicates that brain similarity does not go hand in hand with behavioural similarity, and vice versa.

📄 PDF Abstract BibTeX arXiv:2407.17914

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Focus 설명 없음
ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Similar Papers 제목 키워드 기반

Dual Embodied-Symbolic Concept Representations for Deep Learning

2022-03-01 · Daniel T. Chang

Motivated by recent findings from cognitive neural science, we advocate the use of a dual-level model for concept representations: the embodied level consists of concept-oriented feature representations, and the symbolic…

class-incremental learningClass Incremental LearningDeep LearningFew-Shot Class-Incremental Learning+8

Towards a Theory for Bio - Cyber Physical Systems Modelling

2016-01-26

Currently, CyberPhysical Systems (CPS) represents a great challenge for automatic control and smart systems engineering on both theoretical and practical levels. Designing CPS requires approaches involving multidisciplin…

HVPNet: A Bio-Inspired Network for General Salient and Camouflaged Object Detection

2026-06-30 · Jiawei Xu, Qiangqiang Zhou, Zhouping Li, Yanjiao Shi 외 arxiv

In recent years, most research on multimodal salient object detection (SOD) and camouflaged object detection (COD) typically aims to improve performance through complex cross-modal feature fusion and decoding structures.…

Salient Object Detection

CADgpt: Harnessing Natural Language Processing for 3D Modelling to Enhance Computer-Aided Design Workflows

2024-01-10 · Timo Kapsalis

This paper introduces CADgpt, an innovative plugin integrating Natural Language Processing (NLP) with Rhino3D for enhancing 3D modelling in computer-aided design (CAD) environments. Leveraging OpenAI's GPT-4, CADgpt simp…

Multimodal Grounding for Language Processing

2018-06-17 · COLING 2018 8 · Lisa Beinborn, Teresa Botschen, Iryna Gurevych

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human in…

Survey