Script-Agnostic Language Identification
Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets. However, many modern languages, such as Konkani, Kashmiri, Punjabi etc., are synchronically written in several scripts. Moreover, languages with different writing systems do not share significant lexical, semantic, and syntactic properties in neural representation spaces, which is a disadvantage for closely related languages and low-resource languages, especially those from the Indian Subcontinent. To counter this, we propose learning script-agnostic representations using several different experimental strategies (upscaling, flattening, and script mixing) focusing on four major Dravidian languages (Tamil, Telugu, Kannada, and Malayalam). We find that word-level script randomization and exposure to a language written in multiple scripts is extremely valuable for downstream script-agnostic language identification, while also maintaining competitive performance on naturally occurring text.
Code (1)
Tasks
Language IdentificationSimilar Papers 제목 키워드 기반
Meta-learning-based percussion transcription and $t\bar{a}la$ identification from low-resource audio
This study introduces a meta-learning-based approach for low-resource Tabla Stroke Transcription (TST) and $t\bar{a}la$ identification in Hindustani classical music. Using Model-Agnostic Meta-Learning (MAML), we address …
Drum TranscriptionMeta-LearningMusic TranscriptionTowards Modality-Agnostic Person Re-Identification With Descriptive Query
Person re-identification (ReID) with descriptive query (text or sketch) provides an important supplement for general image-image paradigms, which is usually studied in a single cross-modality matching manner, e.g., t…
DescriptivePerson Re-IdentificationRetrievalGlotScript: A Resource and Tool for Low Resource Writing System Identification
We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than 7,000 languages. It is compiled by aggr…
Language ModelingLanguage ModellingCTC-DID: CTC-Based Arabic dialect identification for streaming applications
This paper proposes a Dialect Identification (DID) approach inspired by the Connectionist Temporal Classification (CTC) loss function as used in Automatic Speech Recognition (ASR). CTC-DID frames the dialect identificati…
Speech RecognitionFrom Prediction to Diagnosis: Reasoning-Aware AI for Photovoltaic Defect Inspection
Reliable photovoltaic defect identification is essential for maintaining energy yield, ensuring warranty compliance, and enabling scalable inspection of rapidly expanding solar fleets. Although recent advances in compute…