Substring Frequency Features for Segmentation of Japanese Katakana Words with Unlabeled Corpora
Word segmentation is crucial in natural language processing tasks for unsegmented languages. In Japanese, many out-of-vocabulary words appear in the phonetic syllabary katakana, making segmentation more difficult due to the lack of clues found in mixed script settings. In this paper, we propose a straightforward approach based on a variant of tf-idf and apply it to the problem of word segmentation in Japanese. Even though our method uses only an unlabeled corpus, experimental results show that it achieves performance comparable to existing methods that use manually labeled corpora. Furthermore, it improves performance of simple word segmentation models trained on a manually labeled corpus.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalMachine TranslationSegmentationSimilar Papers 제목 키워드 기반
A Comparison of Entity Matching Methods between English and Japanese Katakana
Japanese Katakana is one component of the Japanese writing system and is used to express English terms, loanwords, and onomatopoeia in Japanese characters based on the phonemes. The main purpose of this research is to fi…
TransliterationLong Short-Term Memory for Japanese Word Segmentation
This study presents a Long Short-Term Memory (LSTM) neural network approach to Japanese word segmentation (JWS). Previous studies on Chinese word segmentation (CWS) succeeded in using recurrent neural networks such as LS…
Chinese Word SegmentationJapanese Word SegmentationSegmentationDictionary Look-up with Katakana Variant Recognition
The Japanese language has rich variety and quantity of word variant. Since 1980s, it has been recognized that this richness becomes an obstacle against natural language processing. A complete solution, however, has not b…
Morphological AnalysisRetrievalTransliterationCompact and Robust Models for Japanese-English Character-level Machine Translation
Character-level translation has been proved to be able to achieve preferable translation quality without explicit segmentation, but training a character-level model needs a lot of hardware resources. In this paper, we in…
Machine TranslationTranslation