A general tensor-structured compression scheme for efficient large language models

2026-05-24

AI Summary

This work addresses the substantial storage, memory, and computational overhead imposed by dense linear layers in large language models, which hinders efficient deployment and obscures the impact of structural simplifications on model performance. The authors propose Tensor Mixture (MixT), a novel compression method that achieves tensor-structured sparsity at the level of general linear projections for the first time. MixT replaces dense layers with a mixture of natively executable tensor operators, without relying on model-specific components. Leveraging tensor mixture modeling, a unified recovery protocol, and joint entropy and inter-layer geometric analysis, MixT reduces parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1%, and peak memory usage by 60.4% on LLaMA2-7B, while preserving MMLU accuracy. This reveals an underlying relationship between compression ratios and performance phase transitions.

Abstract

Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression.

Problem

• large language models
• tensor compression
• dense linear transformations
• model efficiency
• parameter reduction

Innovation

• tensor-structured compression
• large language models
• linear layer replacement
• model compression
• efficient inference