ECO: Efficient Convolutional Network for Online Video Understanding
The state of the art in video understanding suffers from two problems: (1) The major part of reasoning is performed locally in the video, therefore, it misses important relationships within actions that span several seconds. (2) While there are local methods with fast per-frame processing, the processing of the whole video is not efficient and hampers fast video retrieval or online classification of long-term activities. In this paper, we introduce a network architecture that takes long-term content into account and enables fast per-video processing at the same time. The architecture is based on merging long-term content already in the network rather than in a post-hoc fusion. Together with a sampling strategy, which exploits that neighboring frames are largely redundant, this yields high-quality action classification and video captioning at up to 230 videos per second, where each video can consist of a few hundred frames. The approach achieves competitive performance across all datasets while being 10x to 80x faster than state-of-the-art methods.
Code (6)
Tasks
Action ClassificationAction RecognitionGeneral ClassificationRetrievalVideo CaptioningVideo ClassificationVideo RetrievalVideo UnderstandingSimilar Papers 제목 키워드 기반
Online Video Understanding: OVBench and VideoChat-Online
Multimodal Large Language Models (MLLMs) have significantly progressed in offline video understanding. However, applying these models to real-world scenarios, such as autonomous driving and human-computer interaction, pr…
Autonomous DrivingQuestion AnsweringVideo Question AnsweringVideo UnderstandingOnline Video Super-Resolution with Convolutional Kernel Bypass Graft
Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider th…
Super-ResolutionTransfer LearningVideo Super-ResolutionOVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
Temporal Awareness, the ability to reason dynamically based on the timestamp when a question is raised, is the key distinction between offline and online video LLMs. Unlike offline models, which rely on complete videos f…
BenchmarkingVideo UnderstandingOnline Spatiotemporal Action Detection and Prediction via Causal Representations
In this thesis, we focus on video action understanding problems from an online and real-time processing point of view. We start with the conversion of the traditional offline spatiotemporal action detection pipeline into…
Action DetectionAction RecognitionAction SegmentationAction Understanding+3Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
Benefiting from the advancements in large language models and cross-modal alignment, existing multi-modal video understanding methods have achieved prominent performance in offline scenario. However, online video streams…
cross-modal alignmentLanguage ModellingQuestion AnsweringVideo Question Answering+2