Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We adjust the training distribution of the generalist data with guidance from the limited domain-specific data. We explore several approaches, with clustered importance sampling standing out. This method clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. It is scalable, suitable for pretraining and continued pretraining, it works well in multi-task settings. Our findings demonstrate improvements across different domains in terms of language modeling perplexity and accuracy on multiple-choice question tasks. We also present ablation studies that examine the impact of dataset sizes, clustering configurations, and model sizes.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMultiple-choiceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UBERT: A Novel Language Model for Synonymy Prediction at Scale in the UMLS Metathesaurus
The UMLS Metathesaurus integrates more than 200 biomedical source vocabularies. During the Metathesaurus construction process, synonymous terms are clustered into concepts by human editors, assisted by lexical similarity…
Language ModelingLanguage ModellingPredictionSentenceUBERT: A Novel Language Model for Synonymy Prediction at Scale in the UMLS Metathesaurus
The UMLS Metathesaurus integrates more than 200 biomedical source vocabularies. During the Metathesaurus construction process, synonymous terms are clustered into concepts by human editors, assisted by lexical similarity…
Language ModelingLanguage ModellingPredictionSentencePA-CFL: Privacy-Adaptive Clustered Federated Learning for Transformer-Based Sales Forecasting on Heterogeneous Retail Data
Federated learning (FL) enables retailers to share model parameters for demand forecasting while maintaining privacy. However, heterogeneous data across diverse regions, driven by factors such as varying consumer behavio…
Demand ForecastingFeature ImportanceFederated LearningPrivacy Preserving+1Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization
Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text, enhancing both readability and usability. Despite its importance, the integra…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+3FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment
The growing scale of deep neural networks, encompassing large language models (LLMs) and vision transformers (ViTs), has made training from scratch prohibitively expensive and deployment increasingly costly. These models…