Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian
Addressing the challenge of limited annotated data in specialized fields and low-resource languages is crucial for the effective use of Language Models (LMs). While most Large Language Models (LLMs) are trained on general-purpose English corpora, there is a notable gap in models specifically tailored for Italian, particularly for technical and bureaucratic jargon. This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in these specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. We evaluated the models on downstream tasks such as document classification and entity typing and conducted intrinsic evaluations using Pseudo-Log-Likelihood. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting. Furthermore, the application of calibration techniques and in-domain verbalizers significantly enhances the efficacy of encoder models. These domain-specialized models prove to be particularly advantageous in scenarios where in-domain resources or expertise are scarce. In conclusion, our findings offer new insights into the use of Italian models in specialized contexts, which may have a significant impact on both research and industrial applications in the digital transformation era.
Code (0)
등록된 구현이 없습니다.
Tasks
Document ClassificationEntity TypingGeneral Knowledgezero-shot-classificationZero-Shot LearningSimilar Papers 제목 키워드 기반
Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study
Despite growing interest in automated hate speech detection, most existing approaches overlook the linguistic diversity of online content. Multilingual instruction-tuned large language models such as LLaMA, Aya, Qwen, an…
DiversityHate Speech DetectionAcoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models
Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts.…
Audio ClassificationFew-Shot LearningEmpowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classification
With contrastive pre-training, sentence encoders are generally optimized to locate semantically similar samples closer to each other in their embedding spaces. In this work, we focus on the potential of their embedding s…
ClassificationDescriptiveMultiple-choicePseudo Label+7Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classification
Vision-Language Models for remote sensing have shown promising uses thanks to their extensive pretraining. However, their conventional usage in zero-shot scene classification methods still involves dividing large images …
Scene ClassificationTransductive Zero-Shot Classificationzero-shot-classificationZero-Shot LearningNovel Semantic Prompting for Zero-Shot Action Recognition
Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to han…
Zero-Shot Action RecognitionAction Understanding