LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures

2026-04-22

AI Summary

This study addresses the limited understanding of hierarchical representation dynamics, task-specific knowledge localization, and robustness bottlenecks in diverse large language model architectures, which hinders effective hybrid architecture design and optimization. To this end, we propose LayerTracer, a framework that jointly defines and analyzes “task particles”—layers where target token probabilities exhibit significant increases—and “fragile layers”—those whose output distributions undergo maximal shifts under perturbation. By sequentially extracting hidden states, mapping them to vocabulary probability distributions, and quantifying perturbation effects via masked interventions and Jensen–Shannon divergence, LayerTracer provides an architecture-agnostic analytical tool. Experiments reveal that task particles predominantly emerge in deeper layers and that larger models exhibit stronger layer-wise robustness. The framework enables precise identification of critical layers, offering principled guidance for layer partitioning, module allocation, and gating strategies in hybrid models.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Currently, Large Language Models (LLMs) feature a diversified architectural landscape, including traditional Transformer, GateDeltaNet, and Mamba. However, the evolutionary laws of hierarchical representations, task knowledge formation positions, and network robustness bottleneck mechanisms in various LLM architectures remain unclear, posing core challenges for hybrid architecture design and model optimization. This paper proposes LayerTracer, an architecture-agnostic end-to-end analysis framework compatible with any LLM architecture. By extracting hidden states layer-by-layer and mapping them to vocabulary probability distributions, it achieves joint analysis of task particle localization and layer vulnerability quantification. We define the task particle as the key layer where the target token probability first rises significantly, representing the model's task execution starting point, and the vulnerable layer is defined as the layer with the maximum Jensen-Shannon (JS) divergence between output distributions before and after mask perturbation, reflecting its sensitivity to disturbances. Experiments on models of different parameter scales show that task particles mainly appear in the deep layers of the model regardless of parameter size, while larger-parameter models exhibit stronger hierarchical robustness. LayerTracer provides a scientific basis for layer division, module ratio, and gating switching of hybrid architectures, effectively optimizing model performance. It accurately locates task-effective layers and stability bottlenecks, offering universal support for LLM structure design and interpretability research.

Problem

• Large Language Models
• architectural diversity
• hierarchical representations
• task knowledge localization
• robustness bottleneck

Innovation

• LayerTracer
• task particle
• vulnerable layer
• architecture-agnostic analysis
• hierarchical robustness