Self-Mutual Distillation Learning for Continuous Sign Language Recognition
In recent years, deep learning moves video-based Continuous Sign Language Recognition (CSLR) significantly forward. Currently, a typical network combination for CSLR includes a visual module, which focuses on spatial and short-temporal information, followed by a contextual module, which focuses on long-temporal information, and the Connectionist Temporal Classification (CTC) loss is adopted to train the network. However, due to the limitation of chain rules in back-propagation, the visual module is hard to adjust for seeking optimized visual features. As a result, it enforces that the contextual module focuses on contextual information optimization only rather than balancing efficient visual and contextual information. In this paper, we propose a Self-Mutual Knowledge Distillation (SMKD) method, which enforces the visual and contextual modules to focus on short-term and long-term information and enhances the discriminative power of both modules simultaneously. Specifically, the visual and contextual modules share the weights of their corresponding classifiers, and train with CTC loss simultaneously. Moreover, the spike phenomenon widely exists with CTC loss. Although it can help us choose a few of the key frames of a gloss, it does drop other frames in a gloss and makes the visual feature saturation in the early stage. A gloss segmentation is developed to relieve the spike phenomenon and decrease saturation in the visual module. We conduct experiments on two CSLR benchmarks: PHOENIX14 and PHOENIX14-T. Experimental results demonstrate the effectiveness of the SMKD.
Code (1)
Tasks
Knowledge DistillationSign Language RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
I$^2$MD: 3D Action Representation Learning with Inter- and Intra-modal Mutual Distillation
Recent progresses on self-supervised 3D human action representation learning are largely attributed to contrastive learning. However, in conventional contrastive frameworks, the rich complementarity between different ske…
Contrastive LearningRepresentation LearningCMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation
In 3D action recognition, there exists rich complementary information between skeleton modalities. Nevertheless, how to model and utilize this information remains a challenging problem for self-supervised 3D action repre…
3D Action RecognitionAction RecognitionFew-Shot Skeleton-Based Action RecognitionKnowledge Distillation+2Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability wi…
Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation
Changes in facial expression, head movement, body movement and gesture movement are remarkable cues in sign language recognition, and most of the current continuous sign language recognition(CSLR) research methods mainly…
Sign Language RecognitionMUSE: Feature Self-Distillation with Mutual Information and Self-Information
We present a novel information-theoretic approach to introduce dependency among features of a deep convolutional neural network (CNN). The core idea of our proposed method, called MUSE, is to combine MUtual information a…
image-classificationImage ClassificationKnowledge Distillationobject-detection+1