Context Sensitive Neural Lemmatization with Lematus
The main motivation for developing contextsensitive lemmatizers is to improve performance on unseen and ambiguous words. Yet previous systems have not carefully evaluated whether the use of context actually helps in these cases. We introduce Lematus, a lemmatizer based on a standard encoder-decoder architecture, which incorporates character-level sentence context. We evaluate its lemmatization accuracy across 20 languages in both a full data setting and a lower-resource setting with 10k training examples in each language. In both settings, we show that including context significantly improves results against a context-free version of the model. Context helps more for ambiguous words than for unseen words, though the latter has a greater effect on overall performance differences between languages. We also compare to three previous context-sensitive lemmatization systems, which all use pre-extracted edit trees as well as hand-selected features and/or additional sources of information such as tagged training data. Without using any of these, our context-sensitive model outperforms the best competitor system (Lemming) in the fulldata setting, and performs on par in the lowerresource setting.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLemmatizationMachine TranslationSentenceWord EmbeddingsSimilar Papers 제목 키워드 기반
LemMED: Fast and Effective Neural Morphological Analysis with Short Context Windows
We present LemMED, a character-level encoder-decoder for contextual morphological analysis (combined lemmatization and tagging). LemMED extends and is named after two other attention-based models, namely Lematus, a conte…
DecoderLemmatizationMorphological AnalysisSentenceTraining Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw Text
Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. Using context can help, both for unseen and ambiguous words. Yet most context-sensitive approaches re…
Data AugmentationLEMMALemmatizationSentenceContext Aware Lemmatization and Morphological Tagging Method in Turkish
The smallest part of a word that defines the word is called a word root. Word roots are used to increase success in many applications since they simplify the word. In this study, the lemmatization model, which is a word …
LemmatizationMorphological TaggingContext Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks
We introduce a composite deep neural network architecture for supervised and language independent context sensitive lemmatization. The proposed method considers the task as to identify the correct edit tree representing …
AttributeLEMMALemmatizationVocal Bursts Valence PredictionLemmatization of Multi-word Common Noun Phrases and Named Entities in Polish
In the paper we present a tool for lemmatization of multi-word common noun phrases and named entities for Polish called LemmaPL. The tool is based on a set of manually crafted rules and heuristics utilizing a set of dict…
Lemmatization