A Comparative Study of Pretrained Language Models on Thai Social Text Categorization
The ever-growing volume of data of user-generated content on social media provides a nearly unlimited corpus of unlabeled data even in languages where resources are scarce. In this paper, we demonstrate that state-of-the-art results on two Thai social text categorization tasks can be realized by pretraining a language model on a large noisy Thai social media corpus of over 1.26 billion tokens and later fine-tuned on the downstream classification tasks. Due to the linguistically noisy and domain-specific nature of the content, our unique data preprocessing steps designed for Thai social media were utilized to ease the training comprehension of the model. We compared four modern language models: ULMFiT, ELMo with biLSTM, OpenAI GPT, and BERT. We systematically compared the models across different dimensions including speed of pretraining and fine-tuning, perplexity, downstream classification benchmarks, and performance in limited pretraining data.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationLanguage ModelingLanguage ModellingText CategorizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords
While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often bor…
Language ModelingLanguage ModellingQuantifying fair income distribution in Thailand
Given a vast concern about high income inequality in Thailand as opposed to empirical findings around the world showing people's preference for fair income inequality over unfair income equality, it is therefore importan…
FairnessParsing Thai Social Data: A New Challenge for Thai NLP
Dependency parsing (DP) is a task that analyzes text for syntactic structure and relationship between words. DP is widely used to improve natural language processing (NLP) applications in many languages such as English. …
Dependency ParsingChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning
Large language models (LLMs) exhibit strong natural-language reasoning abilities for clinical decision support, but struggle to effectively model structured longitudinal electronic health records (EHRs). In contrast, EHR…
Multimodal ReasoningThai Universal Dependency Treebank
Automatic dependency parsing of Thai sentences has been underexplored, as evidenced by the lack of large Thai dependency treebanks with complete dependency structures and the lack of a published systematic evaluation of …
Dependency Parsing