SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference
This paper addresses the privacy and security concerns associated with deep neural language models, which serve as crucial components in various modern AI-based applications. These models are often used after being pre-trained and fine-tuned for specific tasks, with deployment on servers accessed through the internet. However, this introduces two fundamental risks: (a) the transmission of user inputs to the server via the network gives rise to interception vulnerabilities, and (b) privacy concerns emerge as organizations that deploy such models store user data with restricted context. To address this, we propose a novel method to adapt and fine-tune transformer-based language models on passkey-encrypted user-specific text. The original pre-trained language model first undergoes a quick adaptation (without any further pre-training) with a series of irreversible transformations applied to the tokenizer and token embeddings. This enables the model to perform inference on encrypted inputs while preventing reverse engineering of text from model parameters and intermediate outputs. After adaptation, models are fine-tuned on encrypted versions of existing training datasets. Experimental evaluation employing adapted versions of renowned models (e.g., BERT, RoBERTa) across established benchmark English and multilingual datasets for text classification and sequence labeling shows that encrypted models achieve performance parity with their original counterparts. This serves to safeguard performance, privacy, and security cohesively.
Code (1)
Tasks
Language Modellingtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Domain Adaptation for Efficiently Fine-tuning Vision Transformer with Encrypted Images
In recent years, deep neural networks (DNNs) trained with transformed data have been applied to various applications such as privacy-preserving learning, access control, and adversarial defenses. However, the use of tran…
Domain AdaptationPrivacy PreservingEfficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision Transformer
We propose a novel method for privacy-preserving deep neural networks (DNNs) with the Vision Transformer (ViT). The method allows us not only to train models and test with visually protected images but to also avoid the …
Domain Adaptationimage-classificationImage ClassificationPrivacy PreservingI can't see it but I can Fine-tune it: On Encrypted Fine-tuning of Transformers using Fully Homomorphic Encryption
In today's machine learning landscape, fine-tuning pretrained transformer models has emerged as an essential technique, particularly in scenarios where access to task-aligned training data is limited. However, challenges…
image-classificationImage ClassificationPrivacy PreservingPrivacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices
We propose a novel method for privacy-preserving fine-tuning vision transformers (ViTs) with encrypted images. Conventional methods using encrypted images degrade model performance compared with that of using plain image…
Privacy PreservingEncrypted extremum seeking for privacy-preserving PID tuning as-a-Service
Wireless communication offers many benefits for control such as substantially reduced deployment costs, higher flexibility, as well as easier data access. It is thus not surprising that smart and wireless sensors and act…
Privacy Preserving