Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts

2024-06-24 · Aditya Sharma, Michael Saxon, William Yang Wang

We present LoCoVQA, a dynamic benchmark generator for evaluating long-context extractive reasoning in vision language models (VLMs). LoCoVQA augments test examples for mathematical reasoning, VQA, and character recognition tasks with increasingly long visual contexts composed of both in-distribution and out-of-distribution distractor images. Across these tasks, a diverse set of VLMs rapidly lose performance as the visual context length grows, often exhibiting a striking logarithmic decay trend. This test assesses how well VLMs can ignore irrelevant information when answering queries -- a task that is quite easy for language models (LMs) in the text domain -- demonstrating that current state-of-the-art VLMs lack this essential capability for many long-context applications.

📄 PDF Abstract BibTeX arXiv:2406.16851

Code (0)

등록된 구현이 없습니다.

Tasks

Mathematical Reasoning Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Exponential Decay Exponential Decay is a learning rate schedule where we decay the learning rate with more iterations using an exponential function: $$ \text{lr} =…

A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks

2022-01-21 · Xiaoyu Ma, Sylvain Sardy, Nick Hengartner, Nikolai Bobenko 외

To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample si…

Finding Needles in Emb(a)dding Haystacks: Legal Document Retrieval via Bagging and SVR Ensembles

2025-01-09 · Kevin Bönisch, Alexander Mehler

We introduce a retrieval approach leveraging Support Vector Regression (SVR) ensembles, bootstrap aggregation (bagging), and embedding spaces on the German Dataset for Legal Information Retrieval (GerDaLIR). By conceptua…

Information Retrieval Retrieval

Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark

2024-07-18 · Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta 외

Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. Howeve…

GPU Image Retrieval Question Answering RAG+4

Needles in Haystacks: On Classifying Tiny Objects in Large Images

2019-08-16 · Nick Pawlowski, Suvrat Bhooshan, Nicolas Ballas, Francesco Ciompi 외

In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classifi…

Classification General Classification image-classification Image Classification+1

Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents

2024-11-23 · CVPR 2025 1 · Jun Chen, Dannong Xu, Junjie Fei, Chun-Mei Feng 외

Large multimodal models (LMMs) have achieved impressive progress in vision-language understanding, yet they face limitations in real-world applications requiring complex reasoning over a large number of images. Existing …

Question Answering RAG Retrieval Retrieval-augmented Generation+1