Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models

2025-05-23

AI Summary

The opaque text generation mechanism of large language models (LLMs) hinders interpretability and controllable editing. Method: We propose the Multi-Scale Probabilistic Generation Theory (MSPGT), the first framework to decouple LLM generation into three semantically distinct scales—global context, intermediate structure, and local token—and dynamically align each scale with corresponding Transformer layer segments. Scale boundaries are rigorously defined via dual metrics: attention span thresholds and inter-layer mutual information peaks, revealing fundamental architectural asymmetries between encoder and decoder in scale specialization. Contribution/Results: Validated across GPT-2, BERT, RoBERTa, and T5 via attention analysis, mutual information quantification, probing tasks, and causal intervention, MSPGT demonstrates strong cross-model stability. Scale-specific interventions significantly modulate lexical diversity (local), syntactic structure (intermediate), and discourse coherence (global), all at *p* < 0.001. The theory enables precise, semantics-aware generation control and targeted editing.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Transformer based language models achieve remarkable performance but remain opaque in how they plan, structure, and realize text. We introduce Multi_Scale Probabilistic Generation Theory (MSPGT), a hierarchical framework that factorizes generation into three semantic scales_global context, intermediate structure, and local word choices and aligns each scale with specific layer ranges in Transformer architectures. To identify scale boundaries, we propose two complementary metrics: attention span thresholds and inter layer mutual information peaks. Across four representative models (GPT-2, BERT, RoBERTa, and T5), these metrics yield stable local/intermediate/global partitions, corroborated by probing tasks and causal interventions. We find that decoder_only models allocate more layers to intermediate and global processing while encoder_only models emphasize local feature extraction. Through targeted interventions, we demonstrate that local scale manipulations primarily influence lexical diversity, intermediate-scale modifications affect sentence structure and length, and global_scale perturbations impact discourse coherence all with statistically significant effects. MSPGT thus offers a unified, architecture-agnostic method for interpreting, diagnosing, and controlling large language models, bridging the gap between mechanistic interpretability and emergent capabilities.

Problem

• Understanding how large language models plan and generate text
• Identifying semantic scales in Transformer layer hierarchies
• Controlling text generation at different semantic scales

Innovation

• Hierarchical framework for multi-scale generation
• Attention span and mutual information metrics
• Targeted interventions at different semantic scales