Partially Recentralization Softmax Loss for Vision-Language Models Robustness
As Large Language Models make a breakthrough in natural language processing tasks (NLP), multimodal technique becomes extremely popular. However, it has been shown that multimodal NLP are vulnerable to adversarial attacks, where the outputs of a model can be dramatically changed by a perturbation to the input. While several defense techniques have been proposed both in computer vision and NLP models, the multimodal robustness of models have not been fully explored. In this paper, we study the adversarial robustness provided by modifying loss function of pre-trained multimodal models, by restricting top K softmax outputs. Based on the evaluation and scoring, our experiments show that after a fine-tuning, adversarial robustness of pre-trained models can be significantly improved, against popular attacks. Further research should be studying, such as output diversity, generalization and the robustness-performance trade-off of this kind of loss functions. Our code will be available after this paper is accepted
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessDiversityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Unbiased Exploration in Partial Label Learning
We consider learning a probabilistic classifier from partially-labelled supervision (inputs denoted with multiple possibilities) using standard neural architectures with a softmax as the final layer. We identify a bias p…
Partial Label LearningDoubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference
Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that l…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+2Partially Mutual Exclusive Softmax for Positive and Unlabeled data
In recent years, softmax together with its fast approximations has become the de-facto loss function for deep neural networks with multiclass predictions. However, softmax is used in many problems that do not fully fit t…
Language ModelingLanguage ModellingLarge-Margin Softmax Loss for Convolutional Neural Networks
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component…
General ClassificationSimCE: Simplifying Cross-Entropy Loss for Collaborative Filtering
The learning objective is integral to collaborative filtering systems, where the Bayesian Personalized Ranking (BPR) loss is widely used for learning informative backbones. However, BPR often experiences slow convergence…
Collaborative FilteringRecommendation Systems