Position: Enough of Scaling LLMs! Lets Focus on Downscaling
We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.
Code (1)
Tasks
PositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Is Haar Enough? Exploring Symlets and Coiflets for Wavelet Convolution Layers
Wavelet convolution layers have recently emerged as an efficient mechanism for enlarging receptive fields through multiresolution analysis, but prior work has fixed the wavelet basis to Haar or Daubechies at a chosen dec…
Semantic SegmentationImage ClassificationA Unified Framework for Knowledge Transfer in Bidirectional Model Scaling
Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small…
Towards New Multiwavelets: Associated Filters and Algorithms. Part I: Theoretical Framework and Investigation of Biomedical Signals, ECG and Coronavirus Cases
Biosignals are nowadays important subjects for scientific researches from both theory and applications especially with the appearance of new pandemics threatening humanity such as the new Coronavirus. One aim in the pres…
NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training…
On The Continuous Steering of the Scale of Tight Wavelet Frames
In analogy with steerable wavelets, we present a general construction of adaptable tight wavelet frames, with an emphasis on scaling operations. In particular, the derived wavelets can be "dilated" by a procedure compara…