Removing Manually-Generated Boilerplate from Electronic Texts: Experiments with Project Gutenberg e-Books
Collaborative work on unstructured or semi-structured documents, such as in literature corpora or source code, often involves agreed upon templates containing metadata. These templates are not consistent across users and over time. Rule-based parsing of these templates is expensive to maintain and tends to fail as new documents are added. Statistical techniques based on frequent occurrences have the potential to identify automatically a large fraction of the templates, thus reducing the burden on the programmers. We investigate the case of the Project Gutenberg corpus, where most documents are in ASCII format with preambles and epilogues that are often copied and pasted or manually typed. We show that a statistical approach can solve most cases though some documents require knowledge of English. We also survey various technical solutions that make our approach applicable to large data sets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Semi-Supervised Method using Gaussian Random Fields for Boilerplate Removal in Web Browsers
Boilerplate removal refers to the problem of removing noisy content from a webpage such as ads and extracting relevant content that can be used by various services. This can be useful in several features in web browsers …
BlockingTranslationExtraction of Relevant Images for Boilerplate Removal in Web Browsers
Boilerplate refers to unwanted and repeated parts of a webpage (such as ads or table of contents) that distracts the user from reading the core content of the webpage, such as a news article. Accurate detection and remov…
A High-Quality Web Corpus of Czech
In our paper, we present main results of the Czech grant project Internet as a Language Corpus, whose aim was to build a corpus of Czech web texts and to develop and publicly release related software tools. Our corpus ma…
ArticlesMachine TranslationPOSPOS Tagging+1M-PACT: An Open Source Platform for Repeatable Activity Classification Research
There are many hurdles that prevent the replication of existing work which hinders the development of new activity classification models. These hurdles include switching between multiple deep learning libraries and the d…
Action ClassificationActivity RecognitionClassificationGeneral ClassificationOn the Salience of Low-Probability Tokens for AI-Generated Text Detection: A Multiscale Uncertainty Perspective
AI-generated text increasingly blends with human writing, raising practical risks such as misinformation, academic misuse, and corpora contamination. While statistical detectors are appealing for efficiency and generaliz…
Text Detection