End-to-End Semantic Video Transformer for Zero-Shot Action Recognition

2022-03-10 · Keval Doshi, Yasin Yilmaz

While video action recognition has been an active area of research for several years, zero-shot action recognition has only recently started gaining traction. In this work, we propose a novel end-to-end trained transformer model which is capable of capturing long range spatiotemporal dependencies efficiently, contrary to existing approaches which use 3D-CNNs. Moreover, to address a common ambiguity in the existing works about classes that can be considered as previously unseen, we propose a new experimentation setup that satisfies the zero-shot learning premise for action recognition by avoiding overlap between the training and testing classes. The proposed approach significantly outperforms the state of the arts in zero-shot action recognition in terms of the the top-1 accuracy on UCF-101, HMDB-51 and ActivityNet datasets. The code and proposed experimentation setup are available in GitHub: https://github.com/Secure-and-Intelligent-Systems-Lab/SemanticVideoTransformer

📄 PDF Abstract BibTeX arXiv:2203.05156

Code (1)

secure-and-intelligent-systems-lab/semanticvideotransformer 공식 구현 pytorch

Tasks

Action Recognition Temporal Action Localization Zero-Shot Action Recognition Zero-Shot Learning

Everything at Once - Multi-Modal Fusion Transformer for Video Retrieval

2022-01-01 · CVPR 2022 1 · Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas 외

Multi-modal learning from video data has seen increased attention recently as it allows training of semantically meaningful embeddings without human annotation, enabling tasks like zero-shot retrieval and action loca…

Action Localization Retrieval Video Retrieval Zero-Shot Video Retrieval

ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection

2023-11-01 · Thinh Phan, Khoa Vo, Duy Le, Gianfranco Doretto 외

Temporal action detection (TAD) involves the localization and classification of action instances within untrimmed videos. While standard TAD follows fully supervised learning with closed-set setting on large training dat…

Action Detection Classification Language Modeling Language Modelling+2

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

2021-12-08 · Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas 외

Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation enabling tasks like zero-shot retrieval and classification. In …

Action Localization Retrieval Video Retrieval Zero-Shot Video Retrieval

Cross-modal Representation Learning for Zero-shot Action Recognition

2022-05-03 · CVPR 2022 1 · Chung-Ching Lin, Kevin Lin, Linjie Li, Lijuan Wang 외

We present a cross-modal Transformer-based framework, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR). Our model employs a conceptually new pipeline by which visual representation…

Action Recognition Representation Learning Zero-Shot Action Recognition

Novel Semantic Prompting for Zero-Shot Action Recognition

2026-03-09 · Salman Iqbal, Waheed Rehman arxiv

Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to han…

Zero-Shot Action Recognition Action Understanding