MVP-BERT: Multi-Vocab Pre-training for Chinese BERT
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT (CITATION), which is based on Chinese characters (chars). Second, the masked language model pre-training is based on a single vocab, limiting its downstream task performances. In this work, we first experimentally demonstrate that building a vocab via Chinese word segmentation (CWS) guided sub-word tokenization (SGT) can improve the performances of Chinese PLMs. Then we propose two versions of multi-vocab pre-training (MVP), Hi-MVP and AL-MVP, to improve the models{'} expressiveness. Experiments show that: (a) MVP training strategies improve PLMs{'} downstream performances, especially it can improve the PLM{'}s performances on span-level tasks; (b) our AL-MVP outperforms the recent AMBERT (CITATION) after large-scale pre-training, and it is more robust against adversarial attacks.
Code (0)
등록된 구현이 없습니다.
Tasks
Chinese Word SegmentationLanguage ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one pr…
Chinese Word SegmentationLanguage ModelingLanguage ModellingSentenceMarkBERT: Marking Word Boundaries Improves Chinese BERT
We present a Chinese BERT model dubbed MarkBERT that uses word information in this work. Existing word-based BERT models regard words as basic units, however, due to the vocabulary limit of BERT, they only cover high-fre…
Chinese Named Entity Recognitionnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+7MarkBERT: Marking Word Boundaries Improves Chinese BERT
We present a Chinese BERT model dubbed MarkBERT that uses word information in this work. Existing word-based BERT models regard words as basic units, however, due to the vocabulary limit of BERT, they only cover high-fre…
Chinese Named Entity Recognitionnamed-entity-recognitionNamed Entity Recognition (NER)POS+6CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
In this paper, we introduce the Chinese corpus from CLUE organization, CLUECorpus2020, a large-scale corpus that can be used directly for self-supervised learning such as pre-training of a language model, or language gen…
8kLanguage ModelingLanguage ModellingSelf-Supervised Learning+1Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words
The standard BERT adopts subword-based tokenization, which may break a word into two or more wordpieces (e.g., converting "lossless" to "loss" and "less"). This will bring inconvenience in following situations: (1) what …
ChunkingCloze TestMachine Reading ComprehensionNatural Language Understanding+4