VideoVista-CulturalLingo: 360$^\circ$ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension
Assessing the video comprehension capabilities of multimodal AI systems can effectively measure their understanding and reasoning abilities. Most video evaluation benchmarks are limited to a single language, typically English, and predominantly feature videos rooted in Western cultural contexts. In this paper, we present VideoVista-CulturalLingo, the first video evaluation benchmark designed to bridge cultural, linguistic, and domain divide in video comprehension. Our work differs from existing benchmarks in the following ways: 1) Cultural diversity, incorporating cultures from China, North America, and Europe; 2) Multi-linguistics, with questions presented in Chinese and English-two of the most widely spoken languages; and 3) Broad domain, featuring videos sourced from hundreds of human-created domains. VideoVista-CulturalLingo contains 1,389 videos and 3,134 QA pairs, and we have evaluated 24 recent open-source or proprietary video large models. From the experiment results, we observe that: 1) Existing models perform worse on Chinese-centric questions than Western-centric ones, particularly those related to Chinese history; 2) Current open-source models still exhibit limitations in temporal understanding, especially in the Event Localization task, achieving a maximum score of only 45.2%; 3) Mainstream models demonstrate strong performance in general scientific questions, while open-source models demonstrate weak performance in mathematics.
Code (1)
Similar Papers 제목 키워드 기반
VideoVista: A Versatile Benchmark for Video Understanding and Reasoning
Despite significant breakthroughs in video analysis driven by the rapid development of large multimodal models (LMMs), there remains a lack of a versatile evaluation benchmark to comprehensively assess these models' perf…
Anomaly DetectionLogical ReasoningObject TrackingVideo UnderstandingDissociated Neuronal Cultures as Model Systems for Self-Organized Prediction
Dissociated neuronal cultures provide a simplified yet effective model system for investigating self-organized prediction and information processing in neural networks. This review consolidates current research demonstra…
Complexity Horizons of Compressed Models in Analog Circuit Analysis
The deployment of Large Language Models (LLMs) for specialized engineering domains, such as circuit analysis, often faces a trade-off between reasoning accuracy and computational efficiency. Traditional evaluation method…
Computational EfficiencyModel CompressionDHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) has made notable progress in complex control tasks by leveraging temporal abstraction. However, previous HRL algorithms often suffer from serious data inefficiency as environment…
Hierarchical Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Empirical Study of Dynamic Regret in Online Model Predictive Control for Linear Time-Varying Systems
Model Predictive Control (MPC) is a widely used technique for managing timevarying systems, supported by extensive theoretical analysis. While theoretical studies employing dynamic regret frameworks have established robu…
Model Predictive ControlPrediction