End-to-End Spatio-Temporal Action Localisation with Video Transformers
The most performant spatio-temporal action localisation models use external person proposals and complex external memory banks. We propose a fully end-to-end, purely-transformer based model that directly ingests an input video, and outputs tubelets -- a sequence of bounding boxes and the action classes at each frame. Our flexible model can be trained with either sparse bounding-box supervision on individual frames, or full tubelet annotations. And in both cases, it predicts coherent tubelets as the output. Moreover, our end-to-end model requires no additional pre-processing in the form of proposals, or post-processing in terms of non-maximal suppression. We perform extensive ablation experiments, and significantly advance the state-of-the-art results on four different spatio-temporal action localisation benchmarks with both sparse keyframes and full tubelet annotations.
Code (0)
등록된 구현이 없습니다.
Tasks
Action DetectionAction RecognitionSpatio-Temporal Action LocalizationSimilar Papers 제목 키워드 기반
Online Real-time Multiple Spatiotemporal Action Localisation and Prediction
We present a deep-learning framework for real-time multiple spatio-temporal (S/T) action localisation, classification and early prediction. Current state-of-the-art approaches work offline and are too slow to be useful i…
Early Action PredictionPredictionSpatio-temporal Human Action Localisation and Instance Segmentation in Temporally Untrimmed Videos
Current state-of-the-art human action recognition is focused on the classification of temporally trimmed videos in which only one action occurs per frame. In this work we address the problem of action localisation and in…
Action RecognitionInstance SegmentationSemantic SegmentationTemporal Action LocalizationSpatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition
Transformer-based methods have recently achieved great advancement on 2D image-based vision tasks. For 3D video-based tasks such as action recognition, however, directly applying spatiotemporal transformers on video data…
Action ClassificationAction RecognitionEnhancing Video Transformers for Action Understanding with VLM-aided Training
Owing to their ability to extract relevant spatio-temporal video embeddings, Vision Transformers (ViTs) are currently the best performing models in video action understanding. However, their generalization over domains o…
Action ClassificationAction RecognitionAction UnderstandingVideoLightFormer: Lightweight Action Recognition using Transformers
Efficient video action recognition remains a challenging problem. One large model after another takes the place of the state-of-the-art on the Kinetics dataset, but real-world efficiency evaluations are often lacking. In…
Action RecognitionTemporal Action Localization