Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene
This paper presents a method for the normalization of historical texts using a combination of weighted finite-state transducers and language models. We have extended our previous work on the normalization of dialectal texts and tested the method against a 17th century literary work in Basque. This preprocessed corpus is made available in the LREC repository. The performance of this method for learning relations between historical and contemporary word forms is evaluated against resources in three languages. The method we present learns to map phonological changes using a noisy channel model. The model is based on techniques commonly used for phonological inference and producing Grapheme-to-Grapheme conversion systems encoded as weighted transducers and produces F-scores above 80{\%} in the task for Basque. A wider evaluation shows that the approach performs equally well with all the languages in our evaluation suite: Basque, Spanish and Slovene. A comparison against other methods that address the same task is also provided.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
To What Extent Does Lexical Normalization Help English-as-a-Second Language Learners to Read Noisy English Texts?
How difficult is it for English-as-a-second language (ESL) learners to read noisy English texts? Do ESL learners need lexical normalization to read noisy English texts? These questions may also affect community formation…
Lexical NormalizationHistorical German Text Normalization Using Type- and Token-Based Language Modeling
Historic variations of spelling poses a challenge for full-text search or natural language processing on historical digitized texts. To minimize the gap between the historic orthography and contemporary spelling, usually…
DecoderLanguage ModelingLanguage ModellingLarge Language Model+2Part-of-Speech Tagging for Historical English
As more historical texts are digitized, there is interest in applying natural language processing tools to these archives. However, the performance of these tools is often unsatisfactory, due to language change and genre…
Domain AdaptationPart-Of-Speech TaggingUnsupervised Domain AdaptationWord Embeddings