Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress
With the advent of Transformer, which was used in translation models in 2017, attention-based architectures began to attract attention. Furthermore, after the emergence of BERT, which strengthened the NLU-specific encoder part, which is a part of the Transformer, and the GPT architecture, which strengthened the NLG-specific decoder part, various methodologies, data, and models for learning the Pretrained Language Model began to appear. Furthermore, in the past three years, various Pretrained Language Models specialized for Korean have appeared. In this paper, we intend to numerically and qualitatively compare and analyze various Korean PLMs released to the public.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModelingLanguage ModellingTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-Eng…
Few-Shot LearningIn-Context LearningPrompt EngineeringKoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional representation learning for natural languages. Due to the lack of pretrained ALBERT models for Korean language, the best available practice is the m…
Representation LearningSentenceAMMU : A Survey of Transformer-based Biomedical Pretrained Language Models
Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL…
Self-Supervised LearningSurveyTransfer LearningA Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation
Recent natural language understanding (NLU) research on the Korean language has been vigorously maturing with the advancements of pretrained language models and datasets. However, Korean pretrained language models still …
Language Model EvaluationLanguage ModelingLanguage ModellingNatural Language Understanding+2GECKO: Generative Language Model for English, Code and Korean
We introduce GECKO, a bilingual large language model (LLM) optimized for Korean and English, along with programming languages. GECKO is pretrained on the balanced, high-quality corpus of Korean and English employing LLaM…
kmmluLanguage ModelingLanguage ModellingLarge Language Model+1