Temporal superimposed crossover module for effective continuous sign language
The ultimate goal of continuous sign language recognition(CSLR) is to facilitate the communication between special people and normal people, which requires a certain degree of real-time and deploy-ability of the model. However, in the previous research on CSLR, little attention has been paid to the real-time and deploy-ability. In order to improve the real-time and deploy-ability of the model, this paper proposes a zero parameter, zero computation temporal superposition crossover module(TSCM), and combines it with 2D convolution to form a "TSCM+2D convolution" hybrid convolution, which enables 2D convolution to have strong spatial-temporal modelling capability with zero parameter increase and lower deployment cost compared with other spatial-temporal convolutions. The overall CSLR model based on TSCM is built on the improved ResBlockT network in this paper. The hybrid convolution of "TSCM+2D convolution" is applied to the ResBlock of the ResNet network to form the new ResBlockT, and random gradient stop and multi-level CTC loss are introduced to train the model, which reduces the final recognition WER while reducing the training memory usage, and extends the ResNet network from image classification task to video recognition task. In addition, this study is the first in CSLR to use only 2D convolution extraction of sign language video temporal-spatial features for end-to-end learning for recognition. Experiments on two large-scale continuous sign language datasets demonstrate the effectiveness of the proposed method and achieve highly competitive results.
Code (1)
Tasks
image-classificationImage ClassificationSign Language RecognitionVideo RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deep Adversarial Decomposition: A Unified Framework for Separating Superimposed Images
Separating individual image layers from a single mixed image has long been an important but challenging task. We propose a unified framework named "deep adversarial decomposition" for single superimposed image separation…
Image Shadow RemovalRain RemovalReflection RemovalShadow RemovalYOLOSA: Object detection based on 2D local feature superimposed self-attention
We analyzed the network structure of real-time object detection models and found that the features in the feature concatenation stage are very rich. Applying an attention module here can effectively improve the detection…
object-detectionObject DetectionReal-Time Object DetectionPets: General Pattern Assisted Architecture For Time Series Analysis
Time series analysis has found widespread applications in areas such as weather forecasting, anomaly detection, and healthcare. However, real-world sequential data often exhibit a superimposed state of various fluctuatio…
Anomaly DetectionImputationTime SeriesTime Series Analysis+1MoCo-EA: Exploiting Adversarial Mode Connectivity for Efficient Evolutionary Attacks
Evolutionary algorithms for adversarial attacks leverage population-based search to discover perturbations without gradient information, but suffer from inefficient crossover operations that destroy adversarial propertie…
Performance Analysis and Improvement of Parallel Differential Evolution
Differential evolution (DE) is an effective global evolutionary optimization algorithm using to solve global optimization problems mainly in a continuous domain. In this field, researchers pay more attention to improving…
global-optimizationMath