Fine-tune your Classifier: Finding Correlations With Temperature
Temperature is a widely used hyperparameter in various tasks involving neural networks, such as classification or metric learning, whose choice can have a direct impact on the model performance. Most of existing works select its value using hyperparameter optimization methods requiring several runs to find the optimal value. We propose to analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations on which we can build a heuristic giving us a default value of temperature. We study the correlation between these extracted statistics and the observed optimal temperatures. This preliminary study on more than a hundred combinations of different datasets and features extractors highlights promising results towards the construction of a general heuristic for temperature.
Code (0)
등록된 구현이 없습니다.
Tasks
Hyperparameter OptimizationMetric LearningSimilar Papers 제목 키워드 기반
Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis
Recent research has revealed that machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. For instance, a sentiment classif…
Classificationtext-classificationText ClassificationYour Large Language Models Are Leaving Fingerprints
It has been shown that finetuned transformers and other supervised detectors effectively distinguish between human and machine-generated text in some situations arXiv:2305.13242, but we find that even simple classifiers …
RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spu…
image-classificationImage Classificationzero-shot-classificationZero-Shot LearningFeeding LLM Annotations to BERT Classifiers at Your Own Risk
Using LLM-generated labels to fine-tune smaller encoder-only models for text classification has gained popularity in various settings. While this approach may be justified in simple and low-stakes applications, we conduc…
text-classificationText ClassificationmultiMentalRoBERTa: A Fine-tuned Multiclass Classifier for Mental Health Disorder
The early detection of mental health disorders from social media text is critical for enabling timely support, risk assessment, and referral to appropriate resources. This work introduces multiMentalRoBERTa, a fine-tuned…