Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3\% performance lost to attacks, raising Recall@1 accuracy from 7.5\% to 19.8\%. On COCO, it recovers 8.1\% performance, improving accuracy from 3.8\% to 11.9\%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength ($α=0.1-0.2$) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Robust Adversarial Defense by Tensor Factorization
As machine learning techniques become increasingly prevalent in data analysis, the threat of adversarial attacks has surged, necessitating robust defense mechanisms. Among these defenses, methods exploiting low-rank appr…
Adversarial DefenseTensorShield: Tensor-based Defense Against Adversarial Attacks on Images
Recent studies have demonstrated that machine learning approaches like deep neural networks (DNNs) are easily fooled by adversarial attacks. Subtle and imperceptible perturbations of the data are able to change the resul…
BIG-bench Machine Learningimage-classificationImage ClassificationTensor DecompositionPDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to gen…
Visual Question AnsweringApplying Tensor Decomposition to image for Robustness against Adversarial Attack
Nowadays the deep learning technology is growing faster and shows dramatic performance in computer vision areas. However, it turns out a deep learning based model is highly vulnerable to some small perturbation called an…
Adversarial AttackDeep LearningTensor DecompositionSampling-Based Decomposition Algorithms for Arbitrary Tensor Networks
We show how to develop sampling-based alternating least squares (ALS) algorithms for decomposition of tensors into any tensor network (TN) format. Provided the TN format satisfies certain mild assumptions, resulting algo…
Tensor DecompositionTensor Networks