Optimised Preprocessing for Automatic Mouth Gesture Classification
Mouth gestures are facial expressions in sign language, that do not refer to lip patterns of a spoken language. Research on this topic has been limited so far. The aim of this work is to automatically classify mouth gestures from video material by training a neural network. This could render time-consuming manual annotation unnecessary and help advance the field of automatic sign language translation. However, it is a challenging task due to the little data available as training material and the similarity of different mouth gesture classes. In this paper we focus on the preprocessing of the data, such as finding the area of the face important for mouth gesture recognition. Furthermore we analyse the duration of mouth gestures and determine the optimal length of video clips for classification. Our experiments show, that this can improve the classification results significantly and helps to reach a near human accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGeneral ClassificationGesture RecognitionSign Language TranslationTranslationSimilar Papers 제목 키워드 기반
Mouthing Recognition with OpenPose in Sign Language
Many avatars focus on the hands and how they express sign language. However, sign language also uses mouth and face gestures to modify verbs, adjectives, or adverbs; these are known as non-manual components of the sign. …
Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL)…
Designing, Playing, and Performing with a Vision-based Mouth Interface
The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn mi…
Mouth Articulation-Based Anchoring for Improved Cross-Corpus Speech Emotion Recognition
Cross-corpus speech emotion recognition (SER) plays a vital role in numerous practical applications. Traditional approaches to cross-corpus emotion transfer often concentrate on adapting acoustic features to align with d…
Cross-corpusEmotion RecognitionSpeech Emotion RecognitionTransfer LearningTransfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language
Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information. This work directly classifies m…
Multi-Task LearningSign Language Recognitionspeech-recognitionSpeech Recognition+2