User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.
Code (1)
Tasks
Lexical NormalizationMorphological AnalysisSimilar Papers 제목 키워드 기반
LCTG Bench: LLM Controlled Text Generation Benchmark
The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructio…
Text GenerationCADEL: A Corpus of Administrative Web Documents for Japanese Entity Linking
Entity linking is the task of associating linguistic expressions with entries in a knowledge base that represent real-world entities and concepts. Language resources for this task have primarily been developed for Englis…
Entity DisambiguationEntity LinkingA Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages
User-generated data sources have gained significance in uncovering Adverse Drug Reactions (ADRs), with an increasing number of discussions occurring in the digital world. However, the existing clinical corpora predominan…
ArticlesAttributePharmacovigilanceCJaFr-v3 : A Freely Available Filtered Japanese-French Aligned Corpus
We present a free Japanese-French parallel corpus. It includes 15M aligned segments and is obtained by compiling and filtering several existing resources. In this paper, we describe the existing resources, their quantity…
Annotating the Focus of Negation in Japanese Text
This paper proposes an annotation scheme for the focus of negation in Japanese text. Negation has its scope and the focus within the scope. The scope of negation is the part of the sentence that is negated; the focus is …
Natural Language InferenceNegationOpinion MiningQuestion Answering+1