Identifying Duplicate and Contradictory Information in Wikipedia
Our study identifies sentences in Wikipedia articles that are either identical or highly similar by applying techniques for near-duplicate detection of web pages. This is accomplished with a MapReduce implementation of minhash to identify clusters of sentences with high Jaccard similarity. We show that these clusters can be categorized into six different types, two of which are particularly interesting: identical sentences quantify the extent to which content in Wikipedia is copied and pasted, and near-duplicate sentences that state contradictory facts point to quality issues in Wikipedia.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesSimilar Papers 제목 키워드 기반
Wiki-Reliability: A Large Scale Dataset for Content Reliability on Wikipedia
Wikipedia is the largest online encyclopedia, used by algorithms and web users as a central hub of reliable information on the web. The quality and reliability of Wikipedia content is maintained by a community of volunte…
ArticlesInformation RetrievalRetrievalIt Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports
When a bug manifests in a user-facing application, it is likely to be exposed through the graphical user interface (GUI). Given the importance of visual information to the process of identifying and understanding such bu…
Optical Character RecognitionOptical Character Recognition (OCR)RetrievalText RetrievalWikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia
Retrieval-augmented generation (RAG) has emerged as a promising solution to mitigate the limitations of large language models (LLMs), such as hallucinations and outdated information. However, it remains unclear how LLMs …
Language ModellingRAGRetrieval-augmented GenerationWorld KnowledgeHow Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP
Wikipedia's perceived high quality and broad language coverage have established it as a fundamental resource in multilingual NLP. In the context of low-resource languages, however, these quality assumptions are increasin…
ArticlesMultilingual NLPWikiContradiction: Detecting Self-Contradiction Articles on Wikipedia
While Wikipedia has been utilized for fact-checking and claim verification to debunk misinformation and disinformation, it is essential to either improve article quality and rule out noisy articles. Self-contradiction is…
ArticlesClaim VerificationFact CheckingMisinformation+1