Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization
Natural Language Video Localization (NLVL), grounding phrases from natural language descriptions to corresponding video segments, is a complex yet critical task in video understanding. Despite ongoing advancements, many existing solutions lack the capability to globally capture temporal dynamics of the video data. In this study, we present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process, based on the input video and language query. The main challenges are the inherent sparsity and discontinuity of a 2D temporal map in devising the diffusion decoder. To address these challenges, we introduce a multi-scale technique and develop an innovative diffusion decoder. Our approach effectively encapsulates the interaction between the query and video data across various time scales. Experiments on the Charades and DiDeMo datasets underscore the potency of our design.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderDenoisingVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MSC: Multi-Scale Spatio-Temporal Causal Attention for Autoregressive Video Diffusion
Diffusion transformers enable flexible generative modeling for video. However, it is still technically challenging and computationally expensive to generate high-resolution videos with rich semantics and complex motion. …
Video GenerationMindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
Autonomous driving has progressed from modular pipelines toward end-to-end unification, and Vision-Language-Action (VLA) models are a natural extension of this journey beyond Vision-to-Action (VA). In practice, driving V…
Continuous ControlAutonomous DrivingLanguage Control Diffusion: Efficiently Scaling through Space, Time, and Tasks
Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have all…
reinforcement-learningReinforcement Learning (RL)Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production
Earlier Sign Language Production (SLP) models typically relied on autoregressive decoding, which naturally preserves temporal causality but suffers from error accumulation at inference time. More recent diffusion-based a…
Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech
Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using …