Ranking Neural Checkpoints
This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.
Code (1)
Tasks
TransferabilityTransfer LearningSimilar Papers 제목 키워드 기반
PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inference
Personalized text-to-image generation lets users fine-tune diffusion models into repositories of concept-specific checkpoints, but serving these repositories efficiently is difficult for two reasons: natural-language req…
Text-to-Image GenerationListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval
We propose ListT5, a novel reranking approach based on Fusion-in-Decoder (FiD) that handles multiple candidate passages at both train and inference time. We also introduce an efficient inference framework for listwise ra…
DecoderRerankingRetrievalHow Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs
We conduct a behavioral exploration of LoRA fine-tuned LLMs for Passage Reranking to understand how relevance signals are learned and deployed by Large Language Models. By fine-tuning Mistral-7B, LLaMA3.1-8B, and Pythia-…
Information RetrievalPassage RerankingRerankingRetrievalThree New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation
Changes to hyperparameters can have a dramatic effect on model accuracy. Thus, the tuning of hyperparameters plays an important role in optimizing machine-learning models. An integral part of the hyperparameter-tuning pr…
Domain AdaptationUnsupervised Domain AdaptationSummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typical…
Abstractive Text SummarizationDocument SummarizationMixture-of-ExpertsRe-Ranking+1