Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models
In-context learning (ICL) performance is known to be sensitive to the prompt design, yet the impact of class label options in zero-shot classification has been largely overlooked. This study presents the first comprehensive empirical study investigating how label option (e.g., lexical choice, order, and elaboration) influences zero-shot ICL classification performance. Our findings reveal that lexical choices for label names (e.g., agree vs.support in stance classification) play an important role, with effects also linked to label orders. An analysis of the model internal states further shows that optimal label names tend to activate fewer outlier neurons in the feed forward network. Based on this observation, we propose Label set Optimization via Activation Distribution kurtosiS (LOADS), a post-hoc approach requiring no gradient propagation. LOADS not only demonstrates effectiveness with only 100 unlabelled samples across different model types and sizes, but also shows cross-lingual transferability.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationIn-Context LearningStance Classificationzero-shot-classificationZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KurTail : Kurtosis-based LLM Quantization
One of the challenges of quantizing a large language model (LLM) is the presence of outliers. Outliers often make uniform quantization schemes less effective, particularly in extreme cases such as 4-bit quantization. We …
GPULanguage ModelingLanguage ModellingLarge Language Model+2Selectivity and Shape in the Design of Forward-Forward Goodness Functions
The Forward-Forward (FF) algorithm trains networks layer-by-layer using a local "goodness function," yet sum-of-squares (SoS) has remained the only choice studied. We systematically explore the goodness-function design s…
Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning
Conventional uncertainty-aware temporal difference (TD) learning often assumes a zero-mean Gaussian distribution for TD errors, leading to inaccurate error representations and compromised uncertainty estimation. We intro…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+1Kurtosis-limited Sphere Shaping for Nonlinear Interference Noise Reduction in Optical Channels
Nonlinear interference (NLI) generated during the propagation of an optical waveform through the fiber depends on the fourth order standardized moment of the channel input distribution, also known as kurtosis. Probabilis…
Statistical properties and multifractality of Bitcoin
Using 1-min returns of Bitcoin prices, we investigate statistical properties and multifractality of a Bitcoin time series. We find that the 1-min return distribution is fat-tailed, and kurtosis largely deviates from the …
Time SeriesTime Series Analysis