A Novel Evaluation Method for Morphological Segmentation
Unsupervised learning of morphological segmentation of words in a language, based only on a large corpus of words, is a challenging task. Evaluation of the learned segmentations is a challenge in itself, due to the inherent ambiguity of the segmentation task. There is no way to posit unique {``}correct{''} segmentation for a set of data in an objective way. Two models may arrive at different ways of segmenting the data, which may nonetheless both be valid. Several evaluation methods have been proposed to date, but they do not insist on consistency of the evaluated model. We introduce a new evaluation methodology, which enforces correctness of segmentation boundaries while also assuring consistency of segmentation decisions across the corpus.
Code (0)
등록된 구현이 없습니다.
Tasks
SegmentationvalidSimilar Papers 제목 키워드 기반
Word Segmentation and Morphological Parsing for Sanskrit
We describe our participation in the Word Segmentation and Morphological Parsing (WSMP) for Sanskrit hackathon. We approach the word segmentation task as a sequence labelling task by predicting edit operations from which…
Morphological AnalysisSegmentationMorphAGram, Evaluation and Framework for Unsupervised Morphological Segmentation
Computational morphological segmentation has been an active research topic for decades as it is beneficial for many natural language processing tasks. With the high cost of manually labeling data for morphology and the i…
SegmentationTowards a First Automatic Unsupervised Morphological Segmentation for Inuinnaqtun
Low-resource polysynthetic languages pose many challenges in NLP tasks, such as morphological analysis and Machine Translation, due to available resources and tools, and the morphologically complex languages. This resear…
Machine TranslationMorphological AnalysisTranslationFrom Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time
The best performing transformer-based language models use subword tokenization techniques, such as Byte-Pair-Encoding (BPE). However, these approaches often overlook linguistic principles, such as morphological segmentat…
Segmentation