Online Temporal Action Localization with Memory-Augmented Transformer
Online temporal action localization (On-TAL) is the task of identifying multiple action instances given a streaming video. Since existing methods take as input only a video segment of fixed size per iteration, they are limited in considering long-term context and require tuning the segment size carefully. To overcome these limitations, we propose memory-augmented transformer (MATR). MATR utilizes the memory queue that selectively preserves the past segment features, allowing to leverage long-term context for inference. We also propose a novel action localization method that observes the current input segment to predict the end time of the ongoing action and accesses the memory queue to estimate the start time of the action. Our method outperformed existing methods on two datasets, THUMOS14 and MUSES, surpassing not only TAL methods in the online setting but also some offline TAL methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Action LocalizationTemporal Action LocalizationSimilar Papers 제목 키워드 기반
HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization
Online video understanding often relies on individual frames, leading to frame-by-frame predictions. Recent advancements such as Online Temporal Action Localization (OnTAL), extend this approach to instance-level predict…
Action LocalizationTemporal Action LocalizationVideo UnderstandingTALLFormer: Temporal Action Localization with a Long-memory Transformer
Most modern approaches in temporal action localization divide this problem into two parts: (i) short-term feature extraction and (ii) long-range temporal boundary localization. Due to the high GPU memory cost caused by p…
Action LocalizationAction RecognitionGPUTemporal Action LocalizationAction Unit Memory Network for Weakly Supervised Temporal Action Localization
Weakly supervised temporal action localization aims to detect and localize actions in untrimmed videos with only video-level labels during training. However, without frame-level annotations, it is challenging to achieve …
Action LocalizationDiversityTemporal Action LocalizationWeakly Supervised Action Localization+1Weakly-Supervised Temporal Action Localization with Bidirectional Semantic Consistency Constraint
Weakly Supervised Temporal Action Localization (WTAL) aims to classify and localize temporal boundaries of actions for the video, given only video-level category labels in the training datasets. Due to the lack of bounda…
Action LocalizationTemporal Action LocalizationWeakly-supervised Temporal Action LocalizationProgress Regression RNN for Online Spatial-Temporal Action Localization in Unconstrained Videos
Previous spatial-temporal action localization methods commonly follow the pipeline of object detection to estimate bounding boxes and labels of actions. However, the temporal relation of an action has not been fully expl…
Action Localizationobject-detectionObject DetectionPosition+2