Tokenization Repair in the Presence of Spelling Errors
We consider the following tokenization repair problem: Given a natural language text with any combination of missing or spurious spaces, correct these. Spelling errors can be present, but it's not part of the problem to correct them. For example, given: "Tispa per isabout token izaionrep air", compute "Tis paper is about tokenizaion repair". We identify three key ingredients of high-quality tokenization repair, all missing from previous work: deep language models with a bidirectional component, training the models on text with spelling errors, and making use of the space information already present. Our methods also improve existing spell checkers by fixing not only more tokenization errors but also more spelling errors: once it is clear which characters form a word, it is much easier for them to figure out the correct word. We provide six benchmarks that cover three use cases (OCR errors, text extraction from PDF, human errors) and the cases of partially correct space information and all spaces missing. We evaluate our methods against the best existing methods and a non-trivial baseline. We provide full reproducibility under https://ad.cs.uni-freiburg.de/publications .
Code (2)
Tasks
Optical Character Recognition (OCR)Spelling CorrectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
C-LLM: Learn to Check Chinese Spelling Errors Character by Character
Chinese Spell Checking (CSC) aims to detect and correct spelling errors in sentences. Despite Large Language Models (LLMs) exhibit robust capabilities and are widely applied in various tasks, their performance on CSC is …
Chinese Spell CheckingLanguage ModelingLanguage ModellingLarge Language ModelCharacter-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Intuitively, human readers cope easily with errors in text; typos, misspelling, word substitutions, etc. do not unduly disrupt natural reading. Previous work indicates that letter transpositions result in increased readi…
BSpell: A CNN-Blended BERT Based Bangla Spell Checker
Bangla typing is mostly performed using English keyboard and can be highly erroneous due to the presence of compound and similarly pronounced letters. Spelling correction of a misspelled word requires understanding of wo…
SentenceSpelling CorrectionThe Influence of Spelling Errors on Content Scoring Performance
Spelling errors occur frequently in educational settings, but their influence on automatic scoring is largely unknown. We therefore investigate the influence of spelling errors on content scoring performance using the ex…
BIG-bench Machine LearningFingerspelling within Sign Language Translation
Fingerspelling poses challenges for sign language processing due to its high-frequency motion and use for open-vocabulary terms. While prior work has studied fingerspelling recognition, there has been little attention to…
Sign Language TranslationTranslation