Evaluating Sentence Segmentation and Word Tokenization Systems on Estonian Web Texts
Texts obtained from web are noisy and do not necessarily follow the orthographic sentence and word boundary rules. Thus, sentence segmentation and word tokenization systems that have been developed on well-formed texts might not perform so well on unedited web texts. In this paper, we first describe the manual annotation of sentence boundaries of an Estonian web dataset and then present the evaluation results of three existing sentence segmentation and word tokenization systems on this corpus: EstNLTK, Stanza and UDPipe. While EstNLTK obtains the highest performance compared to other systems on sentence segmentation on this dataset, the sentence segmentation performance of Stanza and UDPipe remains well below the results obtained on the more well-formed Estonian UD test set.
Code (1)
Tasks
SegmentationSentenceSentence segmentationSimilar Papers 제목 키워드 기반
ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors
In languages without orthographic word boundaries, NLP models perform word segmentation, either as an explicit preprocessing step or as an implicit step in an end-to-end computation. This paper shows that Chinese NLP mod…
SegmentationSentenceSentiment AnalysisEvaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
Tokenization is an important first step in Natural Language Processing (NLP) pipelines because it decides how models learn and represent linguistic information. However, current subword tokenizers like SentencePiece or H…
News ClassificationStochastic Tokenization with a Language Model for Neural Text Classification
For unsegmented languages such as Japanese and Chinese, tokenization of a sentence has a significant impact on the performance of text classification. Sentences are usually segmented with words or subwords by a morpholog…
ClassificationGeneral ClassificationLanguage ModelingLanguage Modelling+5How Important Is Tokenization in French Medical Masked Language Models?
Subword tokenization has become the prevailing standard in the field of natural language processing (NLP) over recent years, primarily due to the widespread utilization of pre-trained language models. This shift began wi…
The SIGMORPHON 2022 Shared Task on Morpheme Segmentation
The SIGMORPHON 2022 shared task on morpheme segmentation challenged systems to decompose a word into a sequence of morphemes and covered most types of morphology: compounds, derivations, and inflections. Subtask 1, word-…
AllMorpheme SegmentaitonSegmentationSentence