TLDR: The research paper introduces CMIF, a Confidential and efficient Model Inference Framework, designed to address privacy and efficiency challenges in Large Language Model (LLM) inference. Traditional methods using Trusted Execution Environments (TEEs) suffer from high latency due to frequent TEE-GPU communication, while Differential Privacy (DP) can degrade LLM performance and semantic understanding. CMIF tackles this by deploying the embedding layer in a client-side TEE and subsequent layers on GPU servers, reducing communication overhead. It also employs an optimized Report-Noisy-Max (RNM) mechanism to sanitize sensitive input tokens by replacing them with semantically similar alternatives, preserving privacy with minimal performance impact. Experiments show CMIF reduces inference overhead and maintains high model performance compared to existing methods.
Large Language Models (LLMs) have become integral to many aspects of our daily lives, from powering autonomous vehicles to assisting with medical diagnoses and machine translation. However, their deployment on cloud servers often raises significant privacy concerns, as personal data, such as sensitive queries, could be collected and used by service providers. Protecting this data during model inference is crucial.
Existing approaches to ensure privacy in LLM inference typically fall into two categories: those using CPU-based Trusted Execution Environments (TEEs) and those employing Differential Privacy (DP). While both have their merits, they also come with notable drawbacks.
TEE-based methods, which aim to secure sensitive operations within isolated hardware environments, often suffer from high inference latency. This is because researchers frequently partition models, offloading linear components to GPUs while keeping nonlinear layers within TEEs. This constant back-and-forth data transmission between TEEs and GPUs, especially for the dense nonlinear layers common in LLMs, creates a significant communication bottleneck and slows down the inference process.
On the other hand, Differential Privacy (DP) approaches protect data by adding random noise. While effective in theory, applying DP to LLMs can compromise their performance and semantic understanding, leading to incoherent or incorrect outputs. Furthermore, many DP methods require access to the LLM’s internal parameters and architecture, which can introduce its own security risks.
To overcome these challenges, a new framework called CMIF (Confidential and efficient Model Inference Framework) has been proposed. CMIF aims to provide dual privacy protection, ensuring both confidentiality and efficiency during LLM inference.
How CMIF Works
CMIF tackles the problem with a two-pronged approach. First, it strategically deploys the LLM’s components. The embedding layer, which is responsible for mapping input tokens into dense vectors and is critical for processing sensitive user data, is securely placed within a client-side TEE, specifically using Intel SGX2 enclaves. This ensures that the most sensitive initial processing occurs in a trusted, isolated environment. The subsequent, more computationally intensive layers of the LLM are then deployed on powerful remote GPU servers. Communication between these components is efficiently managed using PyTorch’s RPC mechanism, minimizing the latency associated with traditional TEE-GPU interactions.
Second, CMIF introduces an optimized Report-Noisy-Max (RNM) sanitization mechanism to protect sensitive user inputs. Instead of relying on encryption that requires frequent TEE decryption, CMIF focuses on sanitizing input tokens before they even reach the main LLM layers on the server. This mechanism identifies low-frequency words as sensitive and replaces them with semantically similar alternatives. It uses a distance function based on cosine similarity to find suitable substitute candidates and then applies a probabilistic mapping function, incorporating Laplace noise, to select the optimal replacement. This process ensures that user data privacy is maintained with only a slight decrease in model performance, as the semantic coherence of the input is largely preserved.
Also Read:
- DP-FedLoRA: Securing On-Device Language Models with Private Federated Fine-Tuning
- ENSI: Boosting Privacy-Preserving AI with Efficient Secure Inference for Large Language Models
Experimental Results
Extensive experiments were conducted using Llama-series models (Llama2-7B and Llama3-8B) and Bert on various natural language processing datasets, including SST-2 and QNLI for text classification, DialogSUM for dialogue summarization, and IFEval for instruction following. The results demonstrated that CMIF significantly reduces the additional inference overhead typically associated with TEEs while effectively preserving user data privacy.
In terms of model performance, CMIF consistently outperformed other differential privacy mechanisms in classification accuracy, especially as the privacy budget (epsilon) increased. This superior performance is attributed to its semantic-based sanitization, which ensures that chosen substitute words are semantically close to the originals. For instance, on the SST-2 dataset, CMIF achieved higher accuracy compared to methods like CUSTEXT and SANTEXT+ at similar privacy levels.
Regarding inference efficiency, CMIF incurred a relatively low additional overhead compared to a CPU+GPU baseline, with the majority of this cost attributed to the SGX2 encryption mechanism. The RNM sanitization mechanism itself accounted for a very small fraction of the total time overhead, demonstrating its efficiency. This indicates that CMIF successfully balances strong privacy protection with practical inference speeds.
A formal security analysis also confirmed that CMIF’s sanitization mechanism satisfies differential privacy guarantees, providing a robust framework for protecting user data.
In conclusion, CMIF offers an innovative and effective solution for confidential and efficient LLM inference. By combining TEE-shielded deployment of the embedding layer with a semantic-preserving Report-Noisy-Max sanitization mechanism, it addresses the limitations of previous approaches, ensuring user data privacy without compromising model performance or incurring significant latency. For more details, you can refer to the full research paper here.
