Keypoint based Sign Language Translation without Glosses
Sign Language Translation (SLT) is a task that has not been studied relatively much compared to the study of Sign Language Recognition (SLR). However, the SLR is a study that recognizes the unique grammar of sign language, which is different from the spoken language and has a problem that non-disabled people cannot easily interpret. So, we're going to solve the problem of translating directly spoken language in sign language video. To this end, we propose a new keypoint normalization method for performing translation based on the skeleton point of the signer and robustly normalizing these points in sign language translation. It contributed to performance improvement by a customized normalization method depending on the body parts. In addition, we propose a stochastic frame selection method that enables frame augmentation and sampling at the same time. Finally, it is translated into the spoken language through an Attention-based translation model. Our method can be applied to various datasets in a way that can be applied to datasets without glosses. In addition, quantitative experimental evaluation proved the excellence of our method.
Code (0)
등록된 구현이 없습니다.
Tasks
Sign Language RecognitionSign Language TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production
This work addresses the challenges associated with the use of glosses in both Sign Language Translation (SLT) and Sign Language Production (SLP). While glosses have long been used as a bridge between sign language and sp…
Gloss-free Sign Language TranslationRepresentation LearningSelf-Supervised LearningSign Language Production+3Better Sign Language Translation with STMC-Transformer
Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to extract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language g…
Sign Language RecognitionSign Language TranslationTranslationMultilingual eXtended WordNet Knowledge Base: Semantic Parsing and Translation of Glosses
This paper presents a method to create WordNet-like lexical resources for different languages. Instead of directly translating glosses from one language to another, we perform first semantic parsing of WordNet glosses an…
Machine TranslationSemantic ParsingTranslationConsiderations for meaningful sign language machine translation based on glosses
Automatic sign language processing is gaining popularity in Natural Language Processing (NLP) research (Yin et al., 2021). In machine translation (MT) in particular, sign language translation based on glosses is a promin…
Machine TranslationSign Language TranslationTranslationSignAttention: On the Interpretability of Transformer Models for Sign Language Translation
This paper presents the first comprehensive interpretability analysis of a Transformer-based Sign Language Translation (SLT) model, focusing on the translation from video-based Greek Sign Language to glosses and text. Le…
Sign Language TranslationTranslation