Weakly-guided Self-supervised Pretraining for Temporal Activity Detection
Temporal Activity Detection aims to predict activity classes per frame, in contrast to video-level predictions in Activity Classification (i.e., Activity Recognition). Due to the expensive frame-level annotations required for detection, the scale of detection datasets is limited. Thus, commonly, previous work on temporal activity detection resorts to fine-tuning a classification model pretrained on large-scale classification datasets (e.g., Kinetics-400). However, such pretrained models are not ideal for downstream detection, due to the disparity between the pretraining and the downstream fine-tuning tasks. In this work, we propose a novel 'weakly-guided self-supervised' pretraining method for detection. We leverage weak labels (classification) to introduce a self-supervised pretext task (detection) by generating frame-level pseudo labels, multi-action frames, and action segments. Simply put, we design a detection task similar to downstream, on large-scale classification data, without extra annotations. We show that the models pretrained with the proposed weakly-guided self-supervised detection task outperform prior work on multiple challenging activity detection benchmarks, including Charades and MultiTHUMOS. Our extensive ablations further provide insights on when and how to use the proposed models for activity detection. Code is available at https://github.com/kkahatapitiya/SSDet.
Code (1)
Tasks
Action DetectionActivity DetectionActivity RecognitionClassificationSimilar Papers 제목 키워드 기반
Weakly Supervised Temporal Sentence Grounding With Uncertainty-Guided Self-Training
The task of weakly supervised temporal sentence grounding aims at finding the corresponding temporal moments of a language description in the video, given video-language correspondence only at video-level. Most exist…
Data AugmentationSentenceTemporal Sentence GroundingWeakly-supervised LearningLearning to diagnose cirrhosis from radiological and histological labels with joint self and weakly-supervised pretraining strategies
Identifying cirrhosis is key to correctly assess the health of the liver. However, the gold standard diagnosis of the cirrhosis needs a medical intervention to obtain the histological confirmation, e.g. the METAVIR score…
Transfer LearningSkeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining
We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target …
Spatio-Temporal Action LocalizationContrastive LearningMulti-pretraining for Large-scale Text Classification
Deep neural network-based pretraining methods have achieved impressive results in many natural language processing tasks including text classification. However, their applicability to large-scale text classification with…
Classificationtext-classificationText Classification`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation
Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rel…
Video Object SegmentationSelf-Supervised Learning