PocketLLM: Ultimate Compression of Large Language Models via Meta Networks

2025-11-19

AI Summary

To address storage and transmission bottlenecks in deploying large language models (LLMs) on edge devices, conventional quantization and pruning methods struggle to balance accuracy and efficiency at high compression ratios. This paper proposes a meta-network-based discrete latent-space compression framework: LLM weights are encoded into discrete latent vectors via a lightweight encoder, then reconstructed using a compact codebook and a small learnable decoder. Only the codebook, decoder parameters, and low-dimensional indices need to be stored—drastically reducing model footprint. Experiments demonstrate that, under a 10× compression ratio, Llama 2-7B incurs negligible accuracy degradation and outperforms state-of-the-art compression approaches. To our knowledge, this is the first method achieving extreme parameter-level compression of LLMs while preserving high fidelity—enabling practical edge deployment without compromising performance.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As Large Language Models (LLMs) continue to grow in size, storing and transmitting them on edge devices becomes increasingly challenging. Traditional methods like quantization and pruning struggle to achieve extreme compression of LLMs without sacrificing accuracy. In this paper, we introduce PocketLLM, a novel approach to compress LLMs in a latent space via meta-networks. A simple encoder network is proposed to project the weights of LLMs into discrete latent vectors, which are then represented using a compact codebook. A lightweight decoder network is employed to map the codebook's representative vectors back to the original weight space. This method allows for significant compression of the large weights in LLMs, consisting solely of a small decoder, a concise codebook, and an index. Extensive experiments show that PocketLLM achieves superior performance even at significantly high compression ratios, e.g., compressing Llama 2-7B by 10x with a negligible drop in accuracy.

Problem

• Compressing large language models for edge device deployment
• Overcoming accuracy loss in extreme model compression
• Reducing storage and transmission requirements for LLMs

Innovation

• Compresses LLMs via meta-networks in latent space
• Projects weights into discrete latent vectors using encoder
• Reconstructs weights using lightweight decoder and codebook