Generating abbreviations using Google Books library
The article describes the original method of creating a dictionary of abbreviations based on the Google Books Ngram Corpus. The dictionary of abbreviations is designed for Russian, yet as its methodology is universal it can be applied to any language. The dictionary can be used to define the function of the period during text segmentation in various applied systems of text processing. The article describes difficulties encountered in the process of its construction as well as the ways to overcome them. A model of evaluating a probability of first and second type errors (extraction accuracy and fullness) is constructed. Certain statistical data for the use of abbreviations are provided.
Code (0)
등록된 구현이 없습니다.
Tasks
Text SegmentationSimilar Papers 제목 키워드 기반
Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution
It is tempting to treat frequency trends from the Google Books data sets as indicators of the "true" popularity of various words and phrases. Doing so allows us to draw quantitatively strong conclusions about the evoluti…
ArticlesMaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resources
Maintenance record logbooks are an emerging text type in NLP. They typically consist of free text documents with many domain specific technical terms, abbreviations, as well as non-standard spelling and grammar, which po…
ClusteringDealing with Abbreviations in the Slovenian Biographical Lexicon
Abbreviations present a significant challenge for NLP systems because they cause tokenization and out-of-vocabulary errors. They can also make the text less readable, especially in reference printed books, where they are…
Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability
Large language models (LLMs) use data to learn about the world in order to produce meaningful correlations and predictions. As such, the nature, scale, quality, and diversity of the datasets used to train these models, o…
Optical Character Recognition (OCR)NLP Tools for Predictive Maintenance Records in MaintNet
Processing maintenance logbook records is an important step in the development of predictive maintenance systems. Logbooks often include free text fields with domain specific terms, abbreviations, and non-standard spelli…
ClusteringPOSPOS Tagging