SLIP: Securing LLMs IP Using Weights Decomposition

2024-07-15

arXiv.org

AI Summary

Protecting intellectual property (IP) of large language models (LLMs) deployed on resource-constrained edge devices remains challenging due to vulnerabilities to model extraction and reverse-engineering attacks. Method: We propose the first provably secure, zero-accuracy-loss, and practical hybrid inference protocol. It leverages matrix decomposition techniques (e.g., SVD/QR) to intelligently shard model weights, enabling a collaborative execution architecture wherein sensitive parameters reside exclusively in a secure (high-cost) enclave while computationally lightweight operations execute on an open (vulnerable) edge device—thereby isolating IP and decoupling computational load. A lightweight encryption-assisted protocol further thwarts inference-time reconstruction attacks. Contribution/Results: We establish the first rigorous security guarantee under zero-accuracy-loss constraints. Experiments across multiple LLMs show no accuracy degradation, end-to-end latency overhead <3%, and parameter reconstruction attack success rate reduced to <0.1%—at par with statistical noise levels—while maintaining compatibility with mainstream edge hardware.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have recently seen widespread adoption, in both academia and industry. As these models grow, they become valuable intellectual property (IP), reflecting enormous investments by their owners. Moreover, the high cost of cloud-based deployment has driven interest towards deployment to edge devices, yet this risks exposing valuable parameters to theft and unauthorized use. Current methods to protect models' IP on the edge have limitations in terms of practicality, loss in accuracy, or suitability to requirements. In this paper, we introduce a novel hybrid inference algorithm, named SLIP, designed to protect edge-deployed models from theft. SLIP is the first hybrid protocol that is both practical for real-world applications and provably secure, while having zero accuracy degradation and minimal impact on latency. It involves partitioning the model between two computing resources, one secure but expensive, and another cost-effective but vulnerable. This is achieved through matrix decomposition, ensuring that the secure resource retains a maximally sensitive portion of the model's IP while performing a minimal amount of computations, and vice versa for the vulnerable resource. Importantly, the protocol includes security guarantees that prevent attackers from exploiting the partition to infer the secured information. Finally, we present experimental results that show the robustness and effectiveness of our method, positioning it as a compelling solution for protecting LLMs.

Problem

• Securing LLM intellectual property from edge device theft
• Preventing model parameter exposure during edge deployment
• Enabling secure hybrid inference with zero accuracy degradation

Innovation

• Hybrid inference algorithm partitions model across resources
• Matrix decomposition protects sensitive IP on secure resource
• Provably secure protocol with zero accuracy degradation overhead