The Diminishing Returns of Masked Language Models to Science

2022-05-23 · Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Kyle Chard, Ian Foster

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770M-parameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model sizes, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if at all, in scientific information extraction tasks and offered possible explanations for the surprising performance differences.

📄 PDF Abstract BibTeX arXiv:2205.11342

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Diminishing Returns in Self-Supervised Learning

2025-12-03 · Oli Bridge, Huey Sun, Botond Branyicskai-Nagy, Charles D'Ornano 외 arxiv

Transformer-based architectures have become a dominant paradigm in vision and language, but their success is often attributed to large model capacity and massive training data. In this work, we examine how self-supervise…

Self-Supervised Learning Semantic Segmentation

How Much is Enough? The Diminishing Returns of Tokenization Training Data

2025-02-27 · Varshini Reddy, Craig W. Schmidt, Yuval Pinter, Chris Tanner

Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization strategy, inference strategy, and training…

Attribute

On the Diminishing Returns of Width for Continual Learning

2024-03-11 · Etash Guha, Vihan Lakshman

While deep neural networks have demonstrated groundbreaking performance in various settings, these models often suffer from \emph{catastrophic forgetting} when trained on new tasks in sequence. Several works have empiric…

Continual Learning Learning Theory

Alethia: A Foundational Encoder for Voice Deepfakes

2026-04-30 · Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti arxiv

Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In …

Zero-shot Generalization DeepFake Detection

Diminishing Returns Shape Constraints for Interpretability and Regularization

2018-12-01 · NeurIPS 2018 12 · Maya Gupta, Dara Bahri, Andrew Cotter, Kevin Canini

We investigate machine learning models that can provide diminishing returns and accelerating returns guarantees to capture prior knowledge or policies about how outputs should depend on inputs. We show that one can buil…

BIG-bench Machine Learning