Squeezing More from Limited Data with Recursive Transformers
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
Large language models (LLMs) are expensive to deploy. Parameter sharing offers a possible path towards reducing their size and cost, but its effectiveness in modern LLMs remains fairly limited. In this work, we revisit "…
Knowledge DistillationJoint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant toke…
Efficient ViTsFeature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by \emph{adversarial examples} that are generated by adding small but purposeful distortions to natural examples. P…
Feature Squeezing Mitigates and Detects Carlini/Wagner Adversarial Examples
Feature squeezing is a recently-introduced framework for mitigating and detecting adversarial examples. In previous work, we showed that it is effective against several earlier methods for generating adversarial examples…
Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
In this work, we present a novel approach to simultaneous knowledge transfer and model compression called \textbf{Weight Squeezing}. With this method, we perform knowledge transfer from a teacher model \textbf{by learnin…
Model Compressiontext-classificationText ClassificationTransfer Learning