CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition
Continuous sign language recognition (CSLR) focuses on interpreting and transcribing sequences of sign language gestures in videos. In this work, we propose CLIP sign language adaptation (CLIP-SLA), a novel CSLR framework that leverages the powerful pre-trained visual encoder from the CLIP model to sign language tasks through parameter-efficient fine-tuning (PEFT). We introduce two variants, SLA-Adapter and SLA-LoRA, which integrate PEFT modules into the CLIP visual encoder, enabling fine-tuning with minimal trainable parameters. The effectiveness of the proposed frameworks is validated on four datasets: Phoenix2014, Phoenix2014-T, CSL-Daily, and Isharah-500, where both CLIP-SLA variants outperformed several SOTA models with fewer trainable parameters. Extensive ablation studies emphasize the effectiveness and flexibility of the proposed methods with different vision-language models for CSLR. These findings showcase the potential of adapting large-scale pre-trained models for scalable and efficient CSLR, which pave the way for future advancements in sign language understanding.
Code (1)
Tasks
parameter-efficient fine-tuningSign Language RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
Vision-language models (VLMs) like CLIP have shown impressive zero-shot and few-shot learning capabilities across diverse applications. However, adapting these models to new fine-grained domains remains difficult due to …
Prompt EngineeringFew-Shot LearningDomain AdaptationRethinking Domain Adaptation and Generalization in the Era of CLIP
In recent studies on domain adaptation, significant emphasis has been placed on the advancement of learning shared knowledge from a source domain to a target domain. Recently, the large vision-language pre-trained model,…
Domain AdaptationZero-Shot LearningFAA-CLIP: Federated Adversarial Adaptation of CLIP
Despite the remarkable performance of vision language models (VLMs) such as Contrastive Language Image Pre-training (CLIP), the large size of these models is a considerable obstacle to their use in federated learning (FL…
Domain AdaptationFederated LearningCLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation
Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting fro…
parameter-efficient fine-tuningDomain AdaptationPoint CloudsIs Less More? Exploring Token Condensation as Training-free Adaptation for CLIP
Contrastive language-image pre-training (CLIP) has shown remarkable generalization ability in image classification. However, CLIP sometimes encounters performance drops on downstream datasets during zero-shot inference. …
image-classificationImage ClassificationTest-time Adaptation