Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

2024-06-13 · Kaavya Rekanar, Martin Hayes, Ganesh Sistu, Ciaran Eising

Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.

📄 PDF Abstract BibTeX arXiv:2406.09203

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Driving Question Answering Visual Question Answering Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

LXMERT LXMERT is a model for learning vision-and-language cross-modality representations. It consists of a Transformer model that consists three encoders: object relationship encoder, a…

Explaining Autonomous Driving Actions with Visual Question Answering

2023-07-19 · Shahin Atakishiyev, Mohammad Salameh, Housam Babiker, Randy Goebel

The end-to-end learning ability of self-driving vehicles has achieved significant milestones over the last decade owing to rapid advances in deep learning and computer vision algorithms. However, as autonomous driving te…

Autonomous Driving Autonomous Vehicles Decision Making Question Answering+3

Towards a performance analysis on pre-trained Visual Question Answering models for autonomous driving

2023-07-18 · Kaavya Rekanar, Ciarán Eising, Ganesh Sistu, Martin Hayes

This short paper presents a preliminary analysis of three popular Visual Question Answering (VQA) models, namely ViLBERT, ViLT, and LXMERT, in the context of answering questions relating to driving scenarios. The perform…

Autonomous Driving Model Selection Question Answering Visual Question Answering+1

Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression

2026-01-11 · Yuliang Cai, Dongqiangzi Ye, Zitian Chen, Chongruo Wu arxiv

Autonomous driving increasingly relies on Visual Question Answering (VQA) to enable vehicles to understand complex surroundings by analyzing visual inputs and textual queries. Currently, a paramount concern for VQA in th…

Visual Question Answering Video Question Answering Computational Efficiency Autonomous Driving

RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Exams

2025-07-25 · Andrei Vlad Man, Răzvan-Alexandru Smădu, Cristian-George Craciun, Dumitru-Clementin Cercel 외 arxiv

The intersection of AI and legal systems presents a growing need for tools that support legal education, particularly in under-resourced languages such as Romanian. In this work, we aim to evaluate the capabilities of La…

Information Retrieval Question Answering

NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving

2025-07-07 · Qucheng Peng, Chen Bai, Guoxiang Zhang, Bo Xu 외 arxiv

Autonomous driving systems have made significant advances in Q&A, perception, prediction, and planning based on local visual information, yet they struggle to incorporate broader navigational context that human drivers r…

Reinforcement Learning Autonomous Driving