An Assessment of the Impact of OCR Noise on Language Models
Neural language models are the backbone of modern-day natural language processing applications. Their use on textual heritage collections which have undergone Optical Character Recognition (OCR) is therefore also increasing. Nevertheless, our understanding of the impact OCR noise could have on language models is still limited. We perform an assessment of the impact OCR noise has on a variety of language models, using data in Dutch, English, French and German. We find that OCR noise poses a significant obstacle to language modelling, with language models increasingly diverging from their noiseless targets as OCR quality lowers. In the presence of small corpora, simpler models including PPMI and Word2Vec consistently outperform transformer-based models in this respect.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingOptical Character RecognitionOptical Character Recognition (OCR)Similar Papers 제목 키워드 기반
Analysis of the influence of final resolution on ADC accuracy
This work is devoted to the study of the influence of quantization noise on the spectral characteristics of a digital signal and the assessment of spectrum measurement errors that arise due to the quantization noise of a…
QuantizationQuakeBERT: Accurate Classification of Social Media Texts for Rapid Earthquake Impact Assessment
Social media aids disaster response but suffers from noise, hindering accurate impact assessment and decision making for resilient cities, which few studies considered. To address the problem, this study proposes the fir…
Disaster ResponseLanguage ModellingLarge Language ModelSentiment AnalysisImpact of internal noise on convolutional neural networks
In this paper, we investigate the impact of noise on a simplified trained convolutional network. The types of noise studied originate from a real optical implementation of a neural network, but we generalize these types …
Theoretical and Numerical Analysis of an Optimal Execution Problem with Uncertain Market Impact
This paper is a continuation of Ishitani and Kato (2015), in which we derived a continuous-time value function corresponding to an optimal execution problem with uncertain market impact as the limit of a discrete-time va…
Assessing the Impact of Conspiracy Theories Using Large Language Models
Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challeng…