Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding
Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified system. However, the scarcity of long videos with detailed captions and precise temporal annotations limits their temporal awareness. In this paper, we propose Seq2Time, a data-oriented training paradigm that leverages sequences of images and short video clips to enhance temporal awareness in long videos. By converting sequence positions into temporal annotations, we transform large-scale image and clip captioning datasets into sequences that mimic the temporal structure of long videos, enabling self-supervised training with abundant time-sensitive data. To enable sequence-to-time knowledge transfer, we introduce a novel time representation that unifies positional information across image sequences, clip sequences, and long videos. Experiments demonstrate the effectiveness of our method, achieving a 27.6% improvement in F1 score and 44.8% in CIDEr on the YouCook2 benchmark and a 14.7% increase in recall on the Charades-STA benchmark compared to the baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
Dense Video CaptioningTransfer LearningVideo CaptioningVideo GroundingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AdapNet: Adaptability Decomposing Encoder-Decoder Network for Weakly Supervised Action Recognition and Localization
The point process is a solid framework to model sequential data, such as videos, by exploring the underlying relevance. As a challenging problem for high-level video understanding, weakly supervised action recognition an…
Action ClassificationAction RecognitionDecoderTemporal Localization+3Transfer of Temporal Logic Formulas in Reinforcement Learning
Transferring high-level knowledge from a source task to a target task is an effective way to expedite reinforcement learning (RL). For example, propositional logic and first-order logic have been used as representations …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Transfer LearningLearning Transferable Spatiotemporal Representations from Natural Script Knowledge
Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years. Despite some progress, existing methods are mostly limited to highly curated dat…
DescriptiveRepresentation LearningVideo UnderstandingLifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines
Continuously learning new tasks using high-level ideas or knowledge is a key capability of humans. In this paper, we propose Lifelong reinforcement learning with Sequential linear temporal logic formulas and Reward Machi…
Lifelong learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledge
Large-scale Video Foundation Models (VFMs) has significantly advanced various video-related tasks, either through task-specific models or Multi-modal Large Language Models (MLLMs). However, the open accessibility of VFMs…
Contrastive LearningAdversarial Attack