Trained on 100 million words and still in shape: BERT meets British National Corpus
While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source -- the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.
Code (2)
Tasks
Language ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words
Pre-trained models are widely used in the tasks of natural language processing nowadays. However, in the specific field of text simplification, the research on improving pre-trained models is still blank. In this work, w…
Language ModelingLanguage ModellingLexical SimplificationMasked Language Modeling+2PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords
While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often bor…
Language ModelingLanguage ModellingPretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words
The standard BERT adopts subword-based tokenization, which may break a word into two or more wordpieces (e.g., converting "lossless" to "loss" and "less"). This will bring inconvenience in following situations: (1) what …
ChunkingCloze TestMachine Reading ComprehensionNatural Language Understanding+4WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of these systems as well. We focus on text--au…
Language ModelingLanguage ModellingAMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
Pre-trained language models such as BERT have exhibited remarkable performances in many tasks in natural language understanding (NLU). The tokens in the models are usually fine-grained in the sense that for languages lik…
Language ModelingLanguage ModellingNatural Language Understanding