Data Augmentation for Sign Language Gloss Translation
Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss-to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low-resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergence by proposing two rule-based heuristics that generate pseudo-parallel gloss-text pairs from monolingual spoken language text. By pre-training on the thus obtained synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationLow Resource NMTMachine TranslationNMTSign Language TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing
Sign language translation from video to spoken text presents unique challenges owing to the distinct grammar, expression nuances, and high variation of visual appearance across different speakers and contexts. The interm…
Data AugmentationSign Language TranslationTranslationCross-modality Data Augmentation for End-to-End Sign Language Translation
End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sig…
Data AugmentationKnowledge DistillationSign Language TranslationText Generation+1Automatic Gloss-level Data Augmentation for Sign Language Translation
Securing sufficient data to enable automatic sign language translation modeling is challenging. The data insufficiency issue exists in both video and text modalities; however, fewer studies have been performed on text da…
Data AugmentationSentenceSign Language TranslationTranslationExplore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation
Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing work…
Data AugmentationSign Language RecognitionSign Language TranslationTranslationIntroducing A Bangla Sentence - Gloss Pair Dataset for Bangla Sign Language Translation and Research
Bangla Sign Language (BdSL) translation represents a low-resource NLP task due to the lack of large-scale datasets that address sentence-level translation. Correspondingly, existing research in this field has been limite…
Sign Language TranslationPrompt Engineering