MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert \cite{devlin2018bert}, which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, \emph{seg\_tok}, to form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models expressiveness. Experiments show that: (a) compared with char based vocabulary, \emph{seg\_tok} does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve \emph{seg\_tok}'s performances on sequence labeling tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Chinese Word SegmentationLanguage ModelingLanguage ModellingSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All
We present a new pre-trained language model (PLM) for modern Hebrew, termed AlephBERTGimmel, which employs a much larger vocabulary (128K items) than standard Hebrew PLMs before. We perform a contrastive analysis of this…
AllLanguage ModelingLanguage ModellingMorphological Analysis+5CMV-BERT: Contrastive multi-vocab pretraining of BERT
In this work, we represent CMV-BERT, which improves the pretraining of a language model via two ingredients: (a) contrastive learning, which is well studied in the area of computer vision; (b) multiple vocabularies, one …
Contrastive LearningLanguage ModelingLanguage ModellingSentenceDunbaaBERT: From Sacrifice to Semantics
Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce Du…
Linguistic AcceptabilityNews ClassificationSentiment AnalysisPre-Training with Whole Word Masking for Chinese BERT
Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trai…
Document ClassificationGeneral ClassificationLanguage ModellingMachine Reading Comprehension+4A Polyphone BERT for Polyphone Disambiguation in Mandarin Chinese
Grapheme-to-phoneme (G2P) conversion is an indispensable part of the Chinese Mandarin text-to-speech (TTS) system, and the core of G2P conversion is to solve the problem of polyphone disambiguation, which is to pick up t…
Polyphone disambiguationtext-to-speechText to Speech