Japanese Text Normalization with Encoder-Decoder Model
Text normalization is the task of transforming lexical variants to their canonical forms. We model the problem of text normalization as a character-level sequence to sequence learning problem and present a neural encoder-decoder model for solving it. To train the encoder-decoder model, many sentences pairs are generally required. However, Japanese non-standard canonical pairs are scarce in the form of parallel corpora. To address this issue, we propose a method of data augmentation to increase data size by converting existing resources into synthesized non-standard forms using handcrafted rules. We conducted an experiment to demonstrate that the synthesized corpus contributes to stably train an encoder-decoder model and improve the performance of Japanese text normalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationDecoderMachine TranslationmodelMorphological AnalysisPart-Of-Speech TaggingText NormalizationSimilar Papers 제목 키워드 기반
Improving Neural Text Normalization with Data Augmentation at Character- and Morphological Levels
In this study, we investigated the effectiveness of augmented data for encoder-decoder-based neural normalization models. Attention based encoder-decoder models are greatly effective in generating many natural languages.…
Data AugmentationDecoderMachine TranslationText Normalization+2Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization
Social media texts, such as tweets from Twitter, contain many types of non-standard tokens, and the number of normalization approaches for handling such noisy text has been increasing. We present a method for automatical…
Machine TranslationMorphological AnalysisText NormalizationUser-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japane…
Lexical NormalizationMorphological AnalysisJapanese Zero Anaphora Resolution Can Benefit from Parallel Texts Through Neural Transfer Learning
Parallel texts of Japanese and a non-pro-drop language have the potential of improving the performance of Japanese zero anaphora resolution (ZAR) because pronouns dropped in the former are usually mentioned explicitly in…
Cross-Lingual TransferDecoderLanguage ModelingLanguage Modelling+3A Text Editing Approach to Joint Japanese Word Segmentation, POS Tagging, and Lexical Normalization
Lexical normalization, in addition to word segmentation and part-of-speech tagging, is a fundamental task for Japanese user-generated text processing. In this paper, we propose a text editing model to solve the three tas…
Japanese Word SegmentationLexical NormalizationPart-Of-Speech TaggingPOS+1