KARIPAP: Quantum-Inspired Tensor Network Compression of Large Language Models Using Infinite Projected Entangled Pair States and Tensor Renormalization Group

2025-10-22

AI Summary

To address the high computational cost, energy consumption, and deployment constraints arising from the massive parameter counts of large language models (LLMs), this work proposes a quantum-inspired tensor network compression framework. We introduce two-dimensional infinite projected entangled pair states (iPEPS) — for the first time applied to LLM compression — to model the multi-directional quantum entanglement structure inherent in attention mechanisms and across Transformer layers, thereby revealing their intrinsic low-dimensional entanglement manifold. Leveraging tensor renormalization group (TRG) methods, we achieve polynomial-time efficient tensor contraction and identify redundant parameters via inter-layer entanglement analysis. Evaluated on LLaMA-2 7B, our method attains 93% memory reduction, 70% parameter pruning, and 50% and 25% speedups in training and inference, respectively, with only a 2–3% accuracy degradation. This establishes a novel paradigm for scalable, energy-efficient, quantum-aware AI architectures.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) like ChatGPT and LLaMA drive rapid progress in generative AI, yet their huge parameter scales create severe computational and environmental burdens. High training costs, energy use, and limited device deployment hinder accessibility. Existing compression - pruning, distillation, low-rank, and quantization - reduces size but ignores complex inter-layer correlations. We propose KARIPAP, a quantum-inspired tensor network compression using Infinite Projected Entangled Pair States (iPEPS) and Tensor Renormalization Group (TRG) contraction. Unlike 1D Matrix Product States, iPEPS captures multi-directional entanglement in attention and deep transformer layers. TRG ensures polynomial-time contraction, making tensorization feasible while preserving key correlation geometry. Experiments on LLaMA-2 7B show up to 93% memory and 70% parameter reduction, with 50% faster training, 25% faster inference, and only 2-3% accuracy loss. Layer-wise entanglement profiling reveals redundancy in deeper layers, confirming their suitability for tensor factorization. KARIPAP demonstrates that modern LLMs occupy low-dimensional entanglement manifolds, enabling scalable, energy-efficient, and quantum-aware AI architectures.

Problem

• Compressing large language models to reduce computational and environmental burdens
• Capturing complex inter-layer correlations ignored by existing compression methods
• Achieving significant parameter reduction while maintaining model accuracy

Innovation

• Quantum-inspired tensor network compression for LLMs
• Uses iPEPS to capture multi-directional layer correlations
• TRG enables polynomial-time contraction preserving geometry