Building a Japanese Typo Dataset from Wikipedia's Revision History
User generated texts contain many typos for which correction is necessary for NLP systems to work. Although a large number of typo{--}correction pairs are needed to develop a data-driven typo correction system, no such dataset is available for Japanese. In this paper, we extract over half a million Japanese typo{--}correction pairs from Wikipedia{'}s revision history. Unlike other languages, Japanese poses unique challenges: (1) Japanese texts are unsegmented so that we cannot simply apply a spelling checker, and (2) the way people inputting kanji logographs results in typos with drastically different surface forms from correct ones. We address them by combining character-based extraction rules, morphological analyzers to guess readings, and various filtering methods. We evaluate the dataset using crowdsourcing and run a baseline seq2seq model for typo correction.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Consistent Classification of Translation Revisions: A Case Study of English-Japanese Student Translations
Consistency is a crucial requirement in text annotation. It is especially important in educational applications, as lack of consistency directly affects learners{'} motivation and learning performance. This paper present…
Decision MakingGeneral Classificationtext annotationTranslationBuilding a Corpus of Manually Revised Texts from Discourse Perspective
This paper presents building a corpus of manually revised texts which includes both before and after-revision information. In order to create such a corpus, we propose a procedure for revising a text from a discourse per…
Mining Naturally-occurring Corrections and Paraphrases from Wikipedia's Revision History
Naturally-occurring instances of linguistic phenomena are important both for training and for evaluating automatic processes on text. When available in large quantities, they also prove interesting material for linguisti…
Resource of Wikipedias in 31 Languages Categorized into Fine-Grained Named Entities
This paper describes a resource of Wikipedias in 31 languages categorized into Extended Named Entity (ENE), which has 219 fine-grained NE categories. We first categorized 920 K Japanese Wikipedia pages according to the E…
AttributeAttribute ExtractionEnsemble LearningLink PredictionHedera: Scalable Indexing and Exploring Entities in Wikipedia Revision History
Much of work in semantic web relying on Wikipedia as the main source of knowledge often work on static snapshots of the dataset. The full history of Wikipedia revisions, while contains much more useful information, is st…
Articles