What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models
Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationLanguage ModelingLanguage ModellingQuantizationSimilar Papers 제목 키워드 기반
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens w…
Code GenerationRational Choice and Artificial Intelligence
The theory of rational choice assumes that when people make decisions they do so in order to maximize their utility. In order to achieve this goal they ought to use all the information available and consider all the choi…
Limits on the accuracy of contact inhibition of locomotion
Cells that collide with each other repolarize away from contact, in a process called contact inhibition of locomotion (CIL), which is necessary for correct development of the embryo. CIL can occur even when cells make a …
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the …
Coexistence of two dengue virus serotypes and forecasting for Madeira island
The first outbreak of dengue occurred in Madeira Island on 2012, featuring one virus serotype. Aedes aegypti was the vector of the disease and it is unlikely that it will be eliminated from the island. Therefore, a new o…
Vocal Bursts Valence Prediction