LinVT: Empower Your Image-level Large Language Model to Understand Videos
Large Language Models (LLMs) have been widely used in various tasks, motivating us to develop an LLM-based assistant for videos. Instead of training from scratch, we propose a module to transform arbitrary well-trained image-based LLMs into video-LLMs (after being trained on video data). To better adapt image-LLMs for processing videos, we introduce two design principles: linear transformation to preserve the original visual-language alignment and representative information condensation from redundant video content. Guided by these principles, we propose a plug-and-play Linear Video Tokenizer(LinVT), which enables existing image-LLMs to understand videos. We benchmark LinVT with six recent visual LLMs: Aquila, Blip-3, InternVL2, Mipha, Molmo and Qwen2-VL, showcasing the high compatibility of LinVT. LinVT-based LLMs achieve state-of-the-art performance across various video benchmarks, illustrating the effectiveness of LinVT in multi-modal video understanding.
Code (1)
Tasks
Language ModelingLanguage ModellingLarge Language ModelVideo Question AnsweringVideo UnderstandingVisual Question AnsweringZero-Shot Video Question AnswerSimilar Papers 제목 키워드 기반
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performa…
Dataset GenerationRAGRetrievalRetrieval-augmented GenerationYourBench: Easy Custom Evaluation Sets for Everyone
Evaluating large language models (LLMs) effectively remains a critical bottleneck, as traditional static benchmarks suffer from saturation and contamination, while human evaluations are costly and slow. This hinders time…
MMLUThe economic value of empowering older patients transitioning from hospital to home: Evidence from the 'Your Care Needs You' intervention
Hospital-to-home transitions are a critical component of effective healthcare delivery, especially for patients aged 75 and older. This study evaluates the cost-effectiveness of the 'Your Care Needs You' (YCNY) intervent…
MMESensitivityFocus Your Distribution: Coarse-to-Fine Non-Contrastive Learning for Anomaly Detection and Localization
The essence of unsupervised anomaly detection is to learn the compact distribution of normal samples and detect outliers as anomalies in testing. Meanwhile, the anomalies in real-world are usually subtle and fine-grained…
Anomaly DetectionContrastive LearningUnsupervised Anomaly DetectionChanging the Environment Based on Empowerment as Intrinsic Motivation
One aspect of intelligence is the ability to restructure your own environment so that the world you live in becomes more beneficial to you. In this paper we investigate how the information-theoretic measure of agent empo…
Decision MakingMinecraft