Segmentation-guided Attention for Visual Question Answering from Remote Sensing Images

2024-07-11 · Lucrezia Tosato, Hichem Boussaid, Flora Weissgerber, Camille Kurtz, Laurent Wendling, Sylvain Lobry

Visual Question Answering for Remote Sensing (RSVQA) is a task that aims at answering natural language questions about the content of a remote sensing image. The visual features extraction is therefore an essential step in a VQA pipeline. By incorporating attention mechanisms into this process, models gain the ability to focus selectively on salient regions of the image, prioritizing the most relevant visual information for a given question. In this work, we propose to embed an attention mechanism guided by segmentation into a RSVQA pipeline. We argue that segmentation plays a crucial role in guiding attention by providing a contextual understanding of the visual information, underlying specific objects or areas of interest. To evaluate this methodology, we provide a new VQA dataset that exploits very high-resolution RGB orthophotos annotated with 16 segmentation classes and question/answer pairs. Our study shows promising results of our new methodology, gaining almost 10% of overall accuracy compared to a classical method on the proposed dataset.

📄 PDF Abstract BibTeX arXiv:2407.08669

Code (0)

등록된 구현이 없습니다.

Tasks

Question Answering Segmentation Visual Question Answering Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Focus 설명 없음

Segmentation Guided Attention Networks for Visual Question Answering

2017-07-01 · ACL 2017 7 · Vasu Sharma, Ankita Bishnu, Labhesh Patel

Common Sense Reasoning Question Answering Scene Understanding Semantic Segmentation+2

SegEQA: Video Segmentation Based Visual Attention for Embodied Question Answering

2019-10-01 · ICCV 2019 10 · Haonan Luo, Guosheng Lin, Zichuan Liu, Fayao Liu 외

Embodied Question Answering (EQA) is a newly defined research area where an agent is required to answer the user's questions by exploring the real world environment. It has attracted increasing research interests due to …

Embodied Question Answering Question Answering Segmentation Video Segmentation+3

ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering

2015-11-18 · Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao 외

We propose a novel attention based deep learning architecture for visual question answering task (VQA). Given an image and an image related natural language question, VQA generates the natural language answer for the que…

Question Answering Visual Question Answering Visual Question Answering (VQA)

Interpretable Visual Question Answering via Reasoning Supervision

2023-09-07 · Maria Parelli, Dimitrios Mallis, Markos Diomataris, Vassilis Pitsikalis

Transformer-based architectures have recently demonstrated remarkable performance in the Visual Question Answering (VQA) task. However, such models are likely to disregard crucial visual cues and often rely on multimodal…

Common Sense Reasoning Question Answering Visual Grounding Visual Question Answering+1

Question-Guided Hybrid Convolution for Visual Question Answering

2018-08-08 · ECCV 2018 9 · Peng Gao, Pan Lu, Hongsheng Li, Shuang Li 외

In this paper, we propose a novel Question-Guided Hybrid Convolution (QGHC) network for Visual Question Answering (VQA). Most state-of-the-art VQA methods fuse the high-level textual and visual features from the neural n…