Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT

2026-06-25

AI Summary

This work addresses the urgent need for high compression ratios in deploying large language models (LLMs) on industrial IoT edge devices, where existing structured pruning methods suffer severe performance degradation under high compression due to the inadequacy of one-time importance estimation. The authors propose a cascaded multi-granularity pruning framework that progressively prunes components in the order of layers, attention heads, and feed-forward channels, interleaved with lightweight low-rank recovery to dynamically reassess component importance. Innovatively guided by information theory, the approach introduces a verifiable Structural Independence Assumption (SIA), revealing for the first time that MHA with GELU satisfies SIA whereas GQA with SwiGLU does not—thereby explaining observed pruning efficacy differences. Evaluated on a bearing fault diagnosis task, the method achieves 13.8× compression on an MHA+GELU architecture with 83.82% accuracy (a 3.70% improvement), reducing inference latency by 67.2% and peak memory usage by 62.5% when deployed on an NVIDIA DGX Spark platform.

Abstract

Deploying large language models (LLMs) on Industrial Internet of Things (IIoT) edge devices demands extreme compression, yet existing structured pruning methods collapse at high compression ratios due to one-shot importance estimation, and their cross-architecture behavior remains unpredictable. This article presents a cascaded multi-granularity pruning framework that removes layers, attention heads, and feed-forward channels in coarse-to-fine order, with lightweight low-rank recovery between stages to re-estimate component importance. An information-theoretic analysis motivates this ordering, and the Structural Independence Assumption (SIA) is formalized as a checkable condition predicting whether per-component pruning criteria are reliable for a given architecture: Multi-Head Attention (MHA)+GELU designs satisfy the SIA, whereas Grouped Query Attention (GQA)+SwiGLU designs violate it. On bearing fault diagnosis spanning 88M to 6.25B-parameter models, the framework extends achievable compression to 13.8 times on MHA+GELU architectures with 83.82% accuracy (+3.70 percentage points (pp) over the strongest baseline), while exposing a ~74pp accuracy collapse on GQA+SwiGLU architectures that violate the SIA. Deployed on an industrial slewing bearing fault diagnosis platform with NVIDIA DGX Spark, compressed models reduce inference latency by up to 67.2% and peak memory by 62.5%, demonstrating viability for IIoT edge inference.

Problem

• structured pruning
• large language models
• Industrial Internet of Things
• model compression
• cross-architecture behavior

Innovation

• cascaded multi-granularity pruning
• structural independence assumption
• on-device LLM inference
• low-rank recovery
• Industrial IoT