Why Inference in Large Models Becomes Decomposable After Training

2026-01-22

AI Summary

Large model inference relies on dense parameter matrices, leading to computational costs and system complexity that scale unsustainably with model size. This work is the first to reveal, from a post-training, model-agnostic structural perspective, that parameter dependencies in large models exhibit strong locality and selectivity: a substantial portion of parameter structures are statistically indistinguishable from their initialization distribution, indicating an inherent decomposability of inference systems. To exploit this property, the authors propose a structure annealing method based on statistical analysis of gradient update events, along with a significance criterion for parameter dependencies, enabling effective identification and removal of unsupported dependencies and extraction of stable, independent substructures. Without altering model functionality, this approach enables structured parallel inference, establishing a new paradigm for efficient large-model inference.

Technology Category

• Machine Learning: Structured Learning
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Inference in large-scale AI models is typically performed on dense parameter matrices, leading to inference cost and system complexity that scale unsustainably with model size. This limitation does not arise from insufficient model capacity, but from treating post-training inference systems as monolithic operators while ignoring internal structures formed during learning. We show that gradient update events in large models are highly localized and selective, leaving many parameter dependencies statistically indistinguishable from their initialization distribution after training. As a result, post-training inference systems are structurally non-uniform and inherently decomposable. Based on this observation, we introduce a post-training statistical criterion and a structural annealing procedure that removes unsupported dependencies and reveals stable, independent substructures. This work establishes a post-training, model-agnostic structural view of inference systems and enables structured, parallel inference without modifying model functionality or interfaces.

Problem

• large models
• inference
• decomposability
• post-training
• structural sparsity

Innovation

• decomposable inference
• post-training structural analysis
• structural annealing
• parameter dependency
• model-agnostic decomposition