An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification
This study investigates the performance of three popular tokenization tools: MeCab, Sudachi, and SentencePiece, when applied as a preprocessing step for sentiment-based text classification of Japanese texts. Using Term Frequency-Inverse Document Frequency (TF-IDF) vectorization, we evaluate two traditional machine learning classifiers: Multinomial Naive Bayes and Logistic Regression. The results reveal that Sudachi produces tokens closely aligned with dictionary definitions, while MeCab and SentencePiece demonstrate faster processing speeds. The combination of SentencePiece, TF-IDF, and Logistic Regression outperforms the other alternatives in terms of classification performance.
Code (1)
Tasks
regressiontext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case st…
Japanese Sentiment Classification using a Tree-Structured Long Short-Term Memory with Attention
Previous approaches to training syntax-based sentiment classification models required phrase-level annotated corpora, which are not readily available in many languages other than English. Thus, we propose the use of tree…
ClassificationGeneral ClassificationSentiment AnalysisSentiment ClassificationJFinTEB: Japanese Financial Text Embedding Benchmark
We introduce JFinTEB, the first comprehensive benchmark specifically designed for evaluating Japanese financial text embeddings. Existing embedding benchmarks provide limited coverage of language-specific and domain-spec…
Sentiment AnalysisText Generationfugashi, a Tool for Tokenizing Japanese in Python
Recent years have seen an increase in the number of large-scale multilingual NLP projects. However, even in such projects, languages with special processing requirements are often excluded. One such language is Japanese.…
Multilingual NLPSiSP: Japanese Situation-dependent Sentiment Polarity Dictionary
In order to deal with the variety of meanings and contexts of words, we created a Japanese Situation-dependent Sentiment Polarity Dictionary (SiSP) of sentiment values labeled for 20 different situations. This dictionary…