Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memory
Long-form video understanding is essential for various applications such as video retrieval, summarizing, and question answering. Yet, traditional approaches demand substantial computing power and are often bottlenecked by GPU memory. To tackle this issue, we present Long-Video Memory Network, Long-VMNet, a novel video understanding method that employs a fixed-size memory representation to store discriminative patches sampled from the input video. Long-VMNet achieves improved efficiency by leveraging a neural sampler that identifies discriminative tokens. Additionally, Long-VMNet only needs one scan through the video, greatly boosting efficiency. Our results on the Rest-ADL dataset demonstrate an 18x -- 75x improvement in inference times for long-form video retrieval and answering questions, with a competitive predictive performance.
Code (0)
등록된 구현이 없습니다.
Tasks
FormGPUQuestion AnsweringRetrievalVideo RetrievalVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SVMnet: Non-parametric image classification based on convolutional SVM ensembles for small training sets
Deep convolutional neural networks (DCNNs) have demonstrated superior power in their ability to classify image data. However, one of the downsides of DCNNs for supervised learning of image data is that their training nor…
image-classificationImage ClassificationVMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation
In recent years, sparse voxel-based methods have become the state-of-the-arts for 3D semantic segmentation of indoor scenes, thanks to the powerful 3D CNNs. Nevertheless, being oblivious to the underlying geometry, voxel…
3D Semantic SegmentationSemantic SegmentationVideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management
Ultra long video understanding remains an open challenge, as existing vision language models (VLMs) falter on such content due to limited context length and inefficient long term memory retention. To address this, recent…
APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or appl…
DVMNet++: Rethinking Relative Pose Estimation for Unseen Objects
Determining the relative pose of a previously unseen object between two images is pivotal to the success of generalizable object pose estimation. Existing approaches typically predict 3D translation utilizing the ground-…
Natural Language UnderstandingObjectPose Estimation