Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding
Visual question answering (VQA) is the multi-modal task of answering natural language questions about an input image. Through cross-dataset adaptation methods, it is possible to transfer knowledge from a source dataset with larger train samples to a target dataset where training set is limited. Suppose a VQA model trained on one dataset train set fails in adapting to another, it is hard to identify the underlying cause of domain mismatch as there could exists a multitude of reasons such as image distribution mismatch and question distribution mismatch. At UCLA, we are working on a VQG module that facilitate in automatically generating OOD shifts that aid in systematically evaluating cross-dataset adaptation capabilities of VQA models.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringQuestion GenerationQuestion-GenerationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or…
Answer GenerationQuestion-Answer-GenerationQuestion AnsweringVisual Question Answering+1FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering
Visual question answering (VQA) systems face significant challenges when adapting to real-world data shifts, especially in multi-modal contexts. While robust fine-tuning strategies are essential for maintaining performan…
BenchmarkingQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)VUDG: A Dataset for Video Understanding Domain Generalization
Video understanding has made remarkable progress in recent years, largely driven by advances in deep models and the availability of large-scale annotated datasets. However, existing works typically ignore the inherent do…
Domain GeneralizationMultiple-choiceQuestion AnsweringSemantic Similarity+3A Fine-Grained Analysis on Distribution Shift
Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and …
HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios
Evaluating the performance of LLMs in multi-turn human-agent interactions presents significant challenges, particularly due to the complexity and variability of user behavior. In this paper, we introduce HammerBench, a n…
Benchmarking