Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning
Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information.
Code (1)
Tasks
In-Context LearningLanguage ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
Revisiting Logit Distributions for Reliable Out-of-Distribution Detection
Out-of-distribution (OOD) detection is critical for ensuring the reliability of deep learning models in open-world applications. While post-hoc methods are favored for their efficiency and ease of deployment, existing ap…
Out-of-Distribution DetectionEnhancing Out-of-Distribution Detection with Extended Logit Normalization
Out-of-distribution (OOD) detection is essential for the safe deployment of machine learning models. Recent advances have explored improved classification losses and representation learning strategies to enhance OOD dete…
Out-of-Distribution DetectionOut of Distribution (OOD) DetectionRepresentation LearningUnleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation
Visual-language models (VLMs) like CLIP exhibit strong generalization but struggle with distribution shifts at test time. Existing training-free test-time adaptation (TTA) methods operate strictly within CLIP's original …
Test-time AdaptationReject Illegal Inputs with Generative Classifier Derived from Any Discriminative Classifier
Generative classifiers have been shown promising to detect illegal inputs including adversarial examples and out-of-distribution samples. Supervised Deep Infomax~(SDIM) is a scalable end-to-end framework to learn generat…
Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
Few-shot Open-set audio classification requires classifying query samples from known classes with a few labeled support samples while rejecting query samples from unknown classes. Transductive inference jointly observes …
Audio Classification