Sanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks
The paper introduces end-to-end neural network models that tokenize Sanskrit by jointly splitting compounds and resolving phonetic merges (Sandhi). Tokenization of Sanskrit depends on local phonetic and distant semantic features that are incorporated using convolutional and recurrent elements. Contrary to most previous systems, our models do not require feature engineering or extern linguistic resources, but operate solely on parallel versions of raw and segmented text. The models discussed in this paper clearly improve over previous approaches to Sanskrit word segmentation. As they are language agnostic, we will demonstrate that they also outperform the state of the art for the related task of German compound splitting.
Code (0)
등록된 구현이 없습니다.
Tasks
Feature EngineeringSimilar Papers 제목 키워드 기반
CharSS: Character-Level Transformer Model for Sanskrit Word Segmentation
Subword tokens in Indian languages inherently carry meaning, and isolating them can enhance NLP tasks, making sub-word segmentation a crucial process. Segmenting Sanskrit and other Indian languages into subtokens is not …
A Dataset for Sanskrit Word Segmentation
The last decade saw a surge in digitisation efforts for ancient manuscripts in Sanskrit. Due to various linguistic peculiarities inherent to the language, even the preliminary tasks such as word segmentation are non-triv…
SegmentationSentenceTransfer LearningLinguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit
The primary focus of this thesis is to make Sanskrit manuscripts more accessible to the end-users through natural language technologies. The morphological richness, compounding, free word orderliness, and low-resource na…
Dependency ParsingMachine TranslationQuestion AnsweringOne Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks
Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morph…
AllDependency ParsingInformation RetrievalLanguage Modelling+4Automatic Speech Recognition in Sanskrit: A New Speech Corpus and Modelling Insights
Automatic speech recognition (ASR) in Sanskrit is interesting, owing to the various linguistic peculiarities present in the language. The Sanskrit language is lexically productive, undergoes euphonic assimilation of phon…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language Modellingspeech-recognition+1