FastQuery: Communication-efficient Embedding Table Query for Private LLM Inference
With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic encryption (HE) has been proposed to protect user query privacy. However, a private embedding table query has to be formulated as a HE-based matrix-vector multiplication problem and suffers from enormous computation and communication overhead. We observe the overhead mainly comes from the neglect of 1) the one-hot nature of user queries and 2) the robustness of the embedding table to low bit-width quantization noise. Hence, in this paper, we propose a private embedding table query optimization framework, dubbed FastQuery. FastQuery features a communication-aware embedding table quantization algorithm and a one-hot-aware dense packing algorithm to simultaneously reduce both the computation and communication costs. Compared to prior-art HE-based frameworks, e.g., Cheetah, Iron, and Bumblebee, FastQuery achieves more than $4.3\times$, $2.7\times$, $1.3\times$ latency reduction, respectively and more than $75.7\times$, $60.2\times$, $20.2\times$ communication reduction, respectively, on both LLAMA-7B and LLAMA-30B.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSimilar Papers 제목 키워드 기반
PrivateMail: Supervised Manifold Learning of Deep Features With Differential Privacy for Image Retrieval
Differential Privacy offers strong guarantees such as immutable privacy under post processing. Thus it is often looked to as a solution to learning on scattered and isolated data. This work focuses on supervised manifold…
Computational EfficiencyContent-Based Image RetrievalImage RetrievalRetrievalDon't forget private retrieval: distributed private similarity search for large language models
While the flexible capabilities of large language models (LLMs) allow them to answer a range of queries based on existing learned knowledge, information retrieval to augment generation is an important tool to allow LLMs …
Information RetrievalRetrievalRetrieval-augmented GenerationData Leakage via Access Patterns of Sparse Features in Deep Learning-based Recommendation Systems
Online personalized recommendation services are generally hosted in the cloud where users query the cloud-based model to receive recommended input such as merchandise of interest or news feed. State-of-the-art recommenda…
Recommendation SystemsProvably-secure symmetric private information retrieval with quantum cryptography
Private information retrieval (PIR) is a database query protocol that provides user privacy, in that the user can learn a particular entry of the database of his interest but his query would be hidden from the data centr…
Information RetrievalRetrievalDifferentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Prompt
Large Language Models (LLMs) have achieved remarkable performance and received significant research interest. The enormous computational demands, however, hinder the local deployment on devices with limited resources. Th…
Natural Language UnderstandingText Generation