Self-Supervised Video Transformers for Isolated Sign Language Recognition
This paper presents an in-depth analysis of various self-supervision methods for isolated sign language recognition (ISLR). We consider four recently introduced transformer-based approaches to self-supervised learning from videos, and four pre-training data regimes, and study all the combinations on the WLASL2000 dataset. Our findings reveal that MaskFeat achieves performance superior to pose-based and supervised video models, with a top-1 accuracy of 79.02% on gloss-based WLASL2000. Furthermore, we analyze these models' ability to produce representations of ASL signs using linear probing on diverse phonological features. This study underscores the value of architecture and pre-training task choices in ISLR. Specifically, our results on WLASL2000 highlight the power of masked reconstruction pre-training, and our linear probing results demonstrate the importance of hierarchical vision transformers for sign language representation.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSign Language RecognitionSimilar Papers 제목 키워드 기반
SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognition
In this paper, we propose a new, simple, and effective Self-supervised Spatio-temporal Transformers (SPARTAN) approach to Group Activity Recognition (GAR) using unlabeled video data. Given a video, we create local and gl…
Activity RecognitionGroup Activity RecognitionSelf-Supervised Video Hashing via Bidirectional Transformers
Most existing unsupervised video hashing methods are built on unidirectional models with less reliable training objectives, which underuse the correlations among frames and the similarity structure between videos. To…
DecoderRetrievalVideo RetrievalLong-Short Temporal Contrastive Learning of Video Transformers
Video transformers have recently emerged as a competitive alternative to 3D CNNs for video understanding. However, due to their large number of parameters and reduced inductive biases, these models require supervised pre…
Action RecognitionContrastive LearningVideo UnderstandingSelf-Supervised Video Hashing with Hierarchical Binary Auto-encoder
Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization mo…
BinarizationDecoderRetrievalVideo RetrievalTime Is MattEr: Temporal Self-supervision for Video Transformers
Understanding temporal dynamics of video is an essential aspect of learning better video representations. Recently, transformer-based architectural designs have been extensively explored for video tasks due to their capa…
Action RecognitionTemporal Action Localization