A Layered Language Model based Hybrid Approach to Automatic Full Diacritization of Arabic
In this paper we present a system for automatic Arabic text diacritization using three levels of analysis granularity in a layered back off manner. We build and exploit diacritized language models (LM) for each of three different levels of granularity: surface form, morphologically segmented into prefix/stem/suffix, and character level. For each of the passes, we use Viterbi search to pick the most probable diacritization per word in the input. We start with the surface form LM, followed by the morphological level, then finally we leverage the character level LM. Our system outperforms all of the published systems evaluated against the same training and test data. It achieves a 10.87{\%} WER for complete full diacritization including lexical and syntactic diacritization, and 3.0{\%} WER for lexical diacritization, ignoring syntactic diacritization.
Code (0)
등록된 구현이 없습니다.
Tasks
Arabic Text DiacritizationFormLanguage ModelingLanguage ModellingMachine TranslationMorphological AnalysisTransliterationWord Sense DisambiguationSimilar Papers 제목 키워드 기반
Diacritic Recognition Performance in Arabic ASR
We present an analysis of diacritic recognition performance in Arabic Automatic Speech Recognition (ASR) systems. As most existing Arabic speech corpora do not contain all diacritical marks, which represent short vowels …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech RecognitionExploiting Arabic Diacritization for High Quality Automatic Annotation
We present a novel technique for Arabic morphological annotation. The technique utilizes diacritization to produce morphological annotations of quality comparable to human annotators. Although Arabic text is generally wr…
LEMMAVocal Bursts Intensity PredictionYAD: Leveraging T5 for Improved Automatic Diacritization of Yorùbá Text
In this work, we present Yor\`ub\'a automatic diacritization (YAD) benchmark dataset for evaluating Yor\`ub\'a diacritization systems. In addition, we pre-train text-to-text transformer, T5 model for Yor\`ub\'a and showe…
Nakdan: Professional Hebrew Diacritizer
We present a system for automatic diacritization of Hebrew text. The system combines modern neural models with carefully curated declarative linguistic knowledge and comprehensive manually constructed tables and dictiona…
Automatic diacritization of Tunisian dialect text using Recurrent Neural Network
The absence of diacritical marks in the Arabic texts generally leads to morphological, syntactic and semantic ambiguities. This can be more blatant when one deals with under-resourced languages, such as the Tunisian dial…