A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of freezing the decoder in a multi-task setup comprising diverse natural language tasks, aiming to reduce deployment overhead and enhance portability to novel tasks. Our experiments, conducted by fine-tuning both individual and multi-task setups on the AlexaTM model, reveal that freezing decoders is highly effective for tasks with natural language outputs and mitigates catastrophic forgetting in multilingual tasks. However, we find that pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks, making it a viable strategy for a broader range of task types.
Code (0)
등록된 구현이 없습니다.
Tasks
Decoderparameter-efficient fine-tuningSimilar Papers 제목 키워드 기반
PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT a…
Image SegmentationVideo SegmentationPrompting Large Language Models for Clinical Temporal Relation Extraction
Objective: This paper aims to prompt large language models (LLMs) for clinical temporal relation extraction (CTRE) in both few-shot and fully supervised settings. Materials and Methods: This study utilizes four LLMs: Enc…
DecoderQuantizationRelationRelation Extraction+1Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates …
DecoderMachine TranslationTranslationMedBLIP: Fine-tuning BLIP for Medical Image Captioning
Medical image captioning is a challenging task that requires generating clinically accurate and semantically meaningful descriptions of radiology images. While recent vision-language models (VLMs) such as BLIP, BLIP2, Ge…
DecoderImage CaptioningLanguages are Modalities: Cross-Lingual Alignment via Encoder Injection
Instruction-tuned Large Language Models (LLMs) underperform on low resource, non-Latin scripts due to tokenizer fragmentation and weak cross-lingual coupling. We present LLINK (Latent Language Injection for Non-English K…