How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristics
What makes a good Large Language Model (LLM)? That it performs well on the relevant benchmarks -- which hopefully measure, with some validity, the presence of capabilities that are also challenged in real application. But what makes the model perform well? What gives a model its abilities? We take a recently introduced type of benchmark that is meant to challenge capabilities in a goal-directed, agentive context through self-play of conversational games, and analyse how performance develops as a function of model characteristics like number of parameters, or type of training. We find that while there is a clear relationship between number of parameters and performance, there is still a wide spread of performance points within a given size bracket, which is to be accounted for by training parameters such as fine-tuning data quality and method. From a more practical angle, we also find a certain degree of unpredictability about performance across access methods, possible due to unexposed sampling parameters, and a, very welcome, performance stability against at least moderate weight quantisation during inference.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
Motivated by many application problems, we consider Markov decision processes (MDPs) with a general loss function and unknown parameters. To mitigate the epistemic uncertainty associated with unknown parameters, we take …
Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes
Neural scaling laws describe how model performance improves as a power law with size, but existing work focuses on models above 100M parameters. The sub-20M regime -- where TinyML and edge AI operate -- remains unexamine…
Analysis of various climate change parameters in India using machine learning
Climate change in India is one of the most alarming problems faced by our community. Due to adverse and sudden changes in climate in past few years, mankind is at threat. Various impacts of climate change include extreme…
BIG-bench Machine LearningregressionConstructing Copulas Using Corrected Hermite Polynomial Expansion for Estimating Cross Foreign Exchange Volatility
Copulas are used to construct joint distributions in many areas. In some problems, it is necessary to deal with correlation structures that are more complicated than the commonly known copulas. A finite order multivariat…
ALCN: Adaptive Local Contrast Normalization
To make Robotics and Augmented Reality applications robust to illumination changes, the current trend is to train a Deep Network with training images captured under many different lighting conditions. Unfortunately, crea…
3D Object DetectionFace Recognitionobject-detectionObject Detection