Improving Document Clustering by Removing Unnatural Language
Technical documents contain a fair amount of unnatural language, such as tables, formulas, and pseudo-code. Unnatural language can bean important factor of confusing existing NLP tools. This paper presents an effective method of distinguishing unnatural language from natural language, and evaluates the impact of un-natural language detection on NLP tasks such as document clustering. We view this problem as an information extraction task and build a multiclass classification model identifying unnatural language components into four categories. First, we create a new annotated corpus by collecting slides and papers in various for-mats, PPT, PDF, and HTML, where unnatural language components are annotated into four categories. We then explore features available from plain text to build a statistical model that can handle any format as long as it is converted into plain text. Our experiments show that re-moving unnatural language components gives an absolute improvement in document cluster-ing by up to 15{\%}. Our corpus and tool are publicly available
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringDocument Layout AnalysisOptical Character Recognition (OCR)Similar Papers 제목 키워드 기반
Improving Document Clustering by Eliminating Unnatural Language
Technical documents contain a fair amount of unnatural language, such as tables, formulas, pseudo-codes, etc. Unnatural language can be an important factor of confusing existing NLP tools. This paper presents an effectiv…
ClusteringSimplifying Multilingual News Clustering Through Projection From a Shared Space
The task of organizing and clustering multilingual news articles for media monitoring is essential to follow news stories in real time. Most approaches to this task focus on high-resource languages (mostly English), with…
ArticlesClusteringCounterfactual Image Networks
We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural i…
counterfactualObjectSegmentationSemantic SegmentationAn enhanced Teaching-Learning-Based Optimization (TLBO) with Grey Wolf Optimizer (GWO) for text feature selection and clustering
Text document clustering can play a vital role in organizing and handling the everincreasing number of text documents. Uninformative and redundant features included in large text documents reduce the effectiveness of the…
ClusteringDimensionality Reductionfeature selectionText ClusteringEfficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation
Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for knowledge integration during large language model (LLM) inference in recent years. However, current RAG implementations face challenges in…
ClusteringHallucinationLanguage ModelingLanguage Modelling+4