Sanskrit Segmentation Revisited
Computationally analyzing Sanskrit texts requires proper segmentation in the initial stages. There have been various tools developed for Sanskrit text segmentation. Of these, G\'erard Huet's Reader in the Sanskrit Heritage Engine analyzes the input text and segments it based on the word parameters - phases like iic, ifc, Pr, Subst, etc., and sandhi (or transition) that takes place at the end of a word with the initial part of the next word. And it enlists all the possible solutions differentiating them with the help of the phases. The phases and their analyses have their use in the domain of sentential parsers. In segmentation, though, they are not used beyond deciding whether the words formed with the phases are morphologically valid. This paper tries to modify the above segmenter by ignoring the phase details (except for a few cases), and also proposes a probability function to prioritize the list of solutions to bring up the most valid solutions at the top.
Code (0)
등록된 구현이 없습니다.
Tasks
SegmentationText SegmentationvalidSimilar Papers 제목 키워드 기반
A Dataset for Sanskrit Word Segmentation
The last decade saw a surge in digitisation efforts for ancient manuscripts in Sanskrit. Due to various linguistic peculiarities inherent to the language, even the preliminary tasks such as word segmentation are non-triv…
SegmentationSentenceTransfer LearningSHR++: An Interface for Morpho-syntactic Annotation of Sanskrit Corpora
We propose a web-based annotation framework, SHR++, for morpho-syntactic annotation of corpora in Sanskrit. SHR++ is designed to generate annotations for the word-segmentation, morphological parsing and dependency analys…
Decision MakingSegmentationvalidOne Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks
Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morph…
AllDependency ParsingInformation RetrievalLanguage Modelling+4Sanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks
The paper introduces end-to-end neural network models that tokenize Sanskrit by jointly splitting compounds and resolving phonetic merges (Sandhi). Tokenization of Sanskrit depends on local phonetic and distant semantic …
Feature EngineeringWord Segmentation in Sanskrit Using Path Constrained Random Walks
In Sanskrit, the phonemes at the word boundaries undergo changes to form new phonemes through a process called as sandhi. A fused sentence can be segmented into multiple possible segmentations. We propose a word segmenta…
SegmentationSentencevalid