How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey
Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computational cost the performance of a model when transferred from a source task to a given target task. Considering the effectiveness of such estimations, the communities of natural language processing also began to study similar problems for the selection of pre-trained language models. However, there is a lack of a comprehensive comparison between these estimation methods yet. Also, the differences between vision and language scenarios make it doubtful whether previous conclusions can be established across fields. In this paper, we first conduct a thorough survey of existing transferability estimation methods being able to find the most suitable model, then we conduct a detailed empirical study for the surveyed methods based on the GLUE benchmark. From qualitative and quantitative analyses, we demonstrate the strengths and weaknesses of existing methods and show that H-Score generally performs well with superiorities in effectiveness and efficiency. We also outline the difficulties of consideration of training details, applicability to text generation, and consistency to certain metrics which shed light on future directions.
Code (1)
Tasks
Language ModelingLanguage ModellingText GenerationSimilar Papers 제목 키워드 기반
Adversarial Training for High-Stakes Reliability
In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses …
Text GenerationVocal Bursts Intensity PredictionLabeling Explicit Discourse Relations using Pre-trained Language Models
Labeling explicit discourse relations is one of the most challenging sub-tasks of the shallow discourse parsing where the goal is to identify the discourse connectives and the boundaries of their arguments. The state-of-…
Discourse ParsingDialects Identification of Armenian Language
The Armenian language has many dialects that differ from each other syntactically, morphologically, and phonetically. In this work, we implement and evaluate models that determine the dialect of a given passage of text. …
Dialect IdentificationLanguage IdentificationSentenceWord EmbeddingsDetecting AI-Generated Images via CLIP
As AI-generated image (AIGI) methods become more powerful and accessible, it has become a critical task to determine if an image is real or AI-generated. Because AIGI lack the signatures of photographs and have their own…
GPUXCompress: LLM assisted Python-based text compression toolkit
This study introduces XCompress, a Python-based tool for effectively utilizing various compression algorithms. XCompress offers manual, brute force, and Large Language Model (LLM) methods to determine the most suitable a…
BenchmarkingLanguage ModelingLanguage ModellingLarge Language Model+1