Finite-state script normalization and processing utilities: The Nisaba Brahmic library
This paper presents an open-source library for efficient low-level processing of ten major South Asian Brahmic scripts. The library provides a flexible and extensible framework for supporting crucial operations on Brahmic scripts, such as NFC, visual normalization, reversible transliteration, and validity checks, implemented in Python within a finite-state transducer formalism. We survey some common Brahmic script issues that may adversely affect the performance of downstream NLP tasks, and provide the rationale for finite-state design and system implementation details.
Code (0)
등록된 구현이 없습니다.
Tasks
SurveyTransliterationSimilar Papers 제목 키워드 기반
Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities
The Brahmic family of scripts is used to record some of the most spoken languages in the world and is arguably the most diverse family of writing systems. In this work, we present several substantial extensions to Brahmi…
TransliterationBeyond Arabic: Software for Perso-Arabic Script Manipulation
This paper presents an open-source software library that provides a set of finite-state transducer (FST) components and corresponding utilities for manipulating the writing systems of languages that use the Perso-Arabic …
TransliterationPositional Description for Numerical Normalization
We present a Positional Description Scheme (PDS) tailored for digit sequences, integrating placeholder value information for each digit. Given the structural limitations of subword tokenization algorithms, language model…
speech-recognitionSpeech RecognitionText Normalizationtext-to-speech+1Latin script keyboards for South Asian languages with finite-state normalization
The use of the Latin script for text entry of South Asian languages is common, even though there is no standard orthography for these languages in the script. We explore several compact finite-state architectures that pe…
TransliterationA Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script
Homophone normalization, where characters that have the same sound in a writing script are mapped to one character, is a pre-processing step applied in Amharic Natural Language Processing (NLP) literature. While this may…
Cross-Lingual TransferMachine TranslationTransfer Learning