홈 › Papers

Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

2020-10-02 · Findings of the Association for Computational Linguistics 2020 · Xiang Dai, Sarvnaz Karimi, Ben Hachey, Cecile Paris

Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0.

📄 PDF Abstract BibTeX arXiv:2010.01150

Code (0)

등록된 구현이 없습니다.

Tasks

Clinical Concept Extraction

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Multi-Head Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Improving Pretraining Data Using Perplexity Correlations

2024-09-09 · Tristan Thrush, Christopher Potts, Tatsunori Hashimoto

Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection expe…

Efficient Pre-Training of LLMs through Truncated SVD Layers

2026-05-27 · Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad, Olivier Francon 외 arxiv

The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computa…

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

2026-08-27 · Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen 외 arxiv

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including op…

MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models

2024-06-10 · Zichun Yu, Spandan Das, Chenyan Xiong

Pretraining data selection has the potential to improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection methods, which rely on either hand-craf…

Language Modelling

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

2025-10-07 · Jie Hao, Rui Yu, Wei Zhang, Huixia Wang 외 arxiv

Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks. However, existing approaches often require leveraging external pr…

Bilevel Optimization