Evaluating historical text normalization systems: How well do they generalize?
We highlight several issues in the evaluation of historical text normalization systems that make it hard to tell how well these systems would actually work in practice---i.e., for new datasets or languages; in comparison to more na\"ive systems; or as a preprocessing step for downstream NLP tools. We illustrate these issues and exemplify our proposed evaluation practices by comparing two neural models against a na\"ive baseline system. We show that the neural models generalize well to unseen words in tests on five languages; nevertheless, they provide no clear benefit over the na\"ive baseline for downstream POS tagging of an English historical collection. We conclude that future work should include more rigorous evaluation, including both intrinsic and extrinsic measures where possible.
Code (0)
등록된 구현이 없습니다.
Tasks
POSPOS TaggingText NormalizationSimilar Papers 제목 키워드 기반
Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene
This paper presents a method for the normalization of historical texts using a combination of weighted finite-state transducers and language models. We have extended our previous work on the normalization of dialectal te…
A Large-Scale Comparison of Historical Text Normalization Systems
There is no consensus on the state-of-the-art approach to historical text normalization. Many techniques have been proposed, including rule-based methods, distance metrics, character-based statistical machine translation…
DecoderMachine TranslationText NormalizationTranslationEvaluating Inter-Annotator Agreement on Historical Spelling Normalization
Improving historical spelling normalization with bi-directional LSTMs and multi-task learning
Natural-language processing of historical documents is complicated by the abundance of variant spellings and lack of annotated data. A common approach is to normalize the spelling of historical words to modern forms. We …
Multi-Task LearningHistorical Text Normalization with Delayed Rewards
Training neural sequence-to-sequence models with simple token-level log-likelihood is now a standard approach to historical text normalization, albeit often outperformed by phrase-based models. Policy gradient training e…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Text Normalization