SlowFast Network for Continuous Sign Language Recognition
The objective of this work is the effective extraction of spatial and dynamic features for Continuous Sign Language Recognition (CSLR). To accomplish this, we utilise a two-pathway SlowFast network, where each pathway operates at distinct temporal resolutions to separately capture spatial (hand shapes, facial expressions) and dynamic (movements) information. In addition, we introduce two distinct feature fusion methods, carefully designed for the characteristics of CSLR: (1) Bi-directional Feature Fusion (BFF), which facilitates the transfer of dynamic semantics into spatial semantics and vice versa; and (2) Pathway Feature Enhancement (PFE), which enriches dynamic and spatial representations through auxiliary subnetworks, while avoiding the need for extra inference time. As a result, our model further strengthens spatial and dynamic representations in parallel. We demonstrate that the proposed framework outperforms the current state-of-the-art performance on popular CSLR datasets, including PHOENIX14, PHOENIX14-T, and CSL-Daily.
Code (1)
Tasks
Sign Language RecognitionSimilar Papers 제목 키워드 기반
Enhanced dynamic sign language recognition using slowfast networks
In this paper, we use the SlowFast Networks developed by the Facebook research team to enhance the accuracy of dynamic sign language recognition. Firstly, we prepared the Word-Level American Sign Language (WLASL) dataset…
Sign Language RecognitionTransfer LearningSlowFast Networks for Video Recognition
We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture moti…
Action ClassificationAction DetectionAction RecognitionAction Recognition In Videos+2Challenge report:VIPriors Action Recognition Challenge
This paper is a brief report to our submission to the VIPriors Action Recognition Challenge. Action recognition has attracted many researchers attention for its full application, but it is still challenging. In this pape…
Action RecognitionAudiovisual SlowFast Networks for Video Recognition
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sou…
Action ClassificationVideo RecognitionOptimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signals
Health-related acoustic signals, such as cough and breathing sounds, are relevant for medical diagnosis and continuous health monitoring. Most existing machine learning approaches for health acoustics are trained and eva…
Contrastive LearningMedical DiagnosisSelf-Supervised Learning