ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs' grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning
Joint audio-video generation aims to synthesize synchronized multisensory content, yet current unified models struggle with fine-grained acoustic control, particularly for identity-preserving speech. Existing approaches …
Video GenerationSelf-Organizing Maps with Variable Input Length for Motif Discovery and Word Segmentation
Time Series Motif Discovery (TSMD) is defined as searching for patterns that are previously unknown and appear with a given frequency in time series. Another problem strongly related with TSMD is Word Segmentation. This …
Language AcquisitionSegmentationTime SeriesTime Series AnalysisGenVC: Self-Supervised Zero-Shot Voice Conversion
Zero-shot voice conversion has recently made substantial progress, but many models still depend on external supervised systems to disentangle speaker identity and linguistic content. Furthermore, current methods often us…
Voice ConversionVTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
Video temporal grounding (VTG) aims to locate specific temporal segments from an untrimmed video based on a linguistic query. Most existing VTG models are trained on extensive annotated video-text pairs, a process that n…
Image CaptioningZero-shot Moment RetrievalGeographic Adaptation of Pretrained Language Models
While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain…
Language IdentificationLanguage ModelingLanguage ModellingMasked Language Modeling+2