Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models
Vision Language Models (VLMs) have been successful at many chart comprehension tasks that require attending to both the images of charts and their accompanying textual descriptions. However, it is not well established how VLM performance profiles map to human-like behaviors. If VLMs can be shown to have human-like chart comprehension abilities, they can then be applied to a broader range of tasks, such as designing and evaluating visualizations for human readers. This paper lays the foundations for such applications by evaluating the accuracy of zero-shot prompting of VLMs on graphical perception tasks with established human performance profiles. Our findings reveal that VLMs perform similarly to humans under specific task and style combinations, suggesting that they have the potential to be used for modeling human performance. Additionally, variations to the input stimuli show that VLM accuracy is sensitive to stylistic changes such as fill color and chart contiguity, even when the underlying data and data mappings are the same.
Code (0)
등록된 구현이 없습니다.
Tasks
Data VisualizationSimilar Papers 제목 키워드 기반
Graphical Perception of Saliency-based Model Explanations
In recent years, considerable work has been devoted to explaining predictive, deep learning-based models, and in turn how to evaluate explanations. An important class of evaluation methods are ones that are human-centere…
Experimental DesignmodelEvaluating Graphical Perception Capabilities of Vision Transformers
Vision Transformers, ViTs, have emerged as a powerful alternative to convolutional neural networks, CNNs, in a variety of image-based tasks. While CNNs have previously been evaluated for their ability to perform graphica…
Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness
Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these mode…
Chart UnderstandingQuestion AnsweringVEIL: How Visual Encoding Hijacking Induces Bias In Vision Models
Rendering time series as chart images for CNN-based classification has become increasingly common in time-series classification (TSC). However, it remains unclear whether models learn underlying temporal patterns or rely…
Towards Understanding Graphical Perception in Large Multimodal Models
Despite the promising results of large multimodal models (LMMs) in complex vision-language tasks that require knowledge, reasoning, and perception abilities together, we surprisingly found that these models struggle with…