Collaborative Temporal Consistency Learning for Point-supervised Natural Language Video Localization
Natural language video localization (NLVL) is a crucial task in video understanding that aims to localize the target moment in videos specified by a given language description. Recently, a point-supervised paradigm has been presented to address this task, requiring only a single annotated frame within the target moment rather than complete temporal boundaries. Compared with the fully-supervised paradigm, it offers a balance between localization accuracy and annotation cost. However, due to the absence of complete annotation, it is challenging to align the video content with language descriptions, consequently hindering accurate moment prediction. To address this problem, we propose a new COllaborative Temporal consistEncy Learning (COTEL) framework that leverages the synergy between saliency detection and moment localization to strengthen the video-language alignment. Specifically, we first design a frame- and a segment-level Temporal Consistency Learning (TCL) module that models semantic alignment across frame saliencies and sentence-moment pairs. Then, we design a cross-consistency guidance scheme, including a Frame-level Consistency Guidance (FCG) and a Segment-level Consistency Guidance (SCG), that enables the two temporal consistency learning paths to reinforce each other mutually. Further, we introduce a Hierarchical Contrastive Alignment Loss (HCAL) to comprehensively align the video and text query. Extensive experiments on two benchmarks demonstrate that our method performs favorably against SoTA approaches. We will release all the source codes.
Code (0)
등록된 구현이 없습니다.
Tasks
Saliency DetectionSentenceVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Ensemling for 3D Point Cloud Domain Adaption
Recently 3D point cloud learning has been a hot topic in computer vision and autonomous driving. Due to the fact that it is difficult to manually annotate a qualitative large-scale 3D point cloud dataset, unsupervised do…
Autonomous DrivingDomain AdaptationSelf-Supervised LearningUnsupervised Domain AdaptationExploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (\textit{i.e.}, labeling only a single frame per action instance) to train a model to effectively locate action instances wit…
Weakly-supervised Temporal Action LocalizationMulti-Task LearningLearning Natural Consistency Representation for Face Forgery Video Detection
Face Forgery videos have elicited critical social public concerns and various detectors have been proposed. However, fully-supervised detectors may lead to easily overfitting to specific forgery methods or videos, and ex…
Representation LearningVideo ClassificationPredNext: Explicit Cross-View Temporal Prediction for Unsupervised Learning in Spiking Neural Networks
Spiking Neural Networks (SNNs), with their temporal processing capabilities and biologically plausible dynamics, offer a natural platform for unsupervised representation learning. However, current unsupervised SNNs predo…
Self-Supervised LearningRepresentation LearningCollaborative Learning for Weakly Supervised Object Detection
Weakly supervised object detection has recently received much attention, since it only requires image-level labels instead of the bounding-box labels consumed in strongly supervised learning. Nevertheless, the save in la…
Objectobject-detectionObject DetectionWeakly Supervised Object Detection