Selected work

Representative Papers

Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning

May 08, 2026

This work investigates how to quantify the influence of data perturbations on observable quantities in Bayesian neural networks and establishes a mapping between data patterns and structural changes. Drawing on linear response theory, the authors define susceptibility via the posterior covariance and construct a susceptibility matrix that serves as the Jacobian of the map from data distribution to structural coordinates. The pseudoinverse of this matrix enables a linear solution to the inverse “patterning” problem, facilitating the design of data perturbations that induce desired structural modifications. By integrating Bayesian inference, the fluctuation–dissipation theorem, and geometric analysis of the loss landscape, this study unifies the theoretical formulations of influence functions and structural susceptibility, provides a computable framework for evaluating sensitivity, and offers an efficient linear approach to controlling neural network behavior.

0 citationsRead paper

Patterning: The Dual of Interpretability

Jan 20, 2026

This work proposes a “patterning” approach that reframes interpretability as the active shaping of a neural network’s internal structure and generalization behavior through deliberate training data design. Grounded in linear response theory, the method employs susceptibilities to quantify model sensitivity to data perturbations and inversely solves for optimal data intervention strategies—such as reweighting and localized learning coefficient optimization—to directionally control the formation of inductive circuits. Experiments on small language models demonstrate that this technique can effectively accelerate or delay the emergence of specific circuits and, in the context of a bracket balancing task, guide the model to learn a prescribed algorithm. This represents the first demonstration of actively writing and regulating internal model structures through targeted data interventions.

0 citationsRead paper

Towards Spectroscopy: Susceptibility Clusters in Language Models

Jan 19, 2026

This study addresses the need to uncover the internal structure of language models and their response mechanisms to input perturbations. To this end, it introduces spectroscopic principles into language model analysis and proposes a susceptibility-based clustering method. By perturbing the distribution of context tokens and employing Stochastic Gradient Langevin Dynamics (SGLD) to approximate the local Gibbs posterior, the approach combines admittance clustering with covariance analysis to identify semantic units organized by similar causal mechanisms within the model. Applied to Pythia-14M, the method successfully discovers 510 interpretable clusters capturing patterns in syntax, code, and mathematical notation, with 50% showing correspondence to features extracted by sparse autoencoders, thereby validating both the efficacy and interpretability of the proposed framework.

0 citationsRead paper

Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

Oct 13, 2025

This study addresses the theoretical assessment of neural network compressibility limits. We extend the Minimum Description Length (MDL) principle—traditionally applicable only to regular models—to singular statistical models by integrating singular learning theory, and propose a novel model complexity estimator based on the Local Learning Coefficient (LLC). Systematic evaluation on the Pythia model family demonstrates a strong linear correlation between LLC and achievable compression ratios across diverse techniques, including quantization and tensor decomposition. Our approach yields the first computationally tractable, theoretically grounded complexity measure for neural networks, overcoming the fundamental limitation that conventional MDL is inapplicable to non-regular (singular) models. By establishing a principled, interpretable link between intrinsic model complexity and compressibility, this work provides a rigorous theoretical framework for characterizing fundamental compression limits in deep learning.

0 citationsRead paper

Recent publications

Latest Papers

Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning

May 08, 2026

This work investigates how to quantify the influence of data perturbations on observable quantities in Bayesian neural networks and establishes a mapping between data patterns and structural changes. Drawing on linear response theory, the authors define susceptibility via the posterior covariance and construct a susceptibility matrix that serves as the Jacobian of the map from data distribution to structural coordinates. The pseudoinverse of this matrix enables a linear solution to the inverse “patterning” problem, facilitating the design of data perturbations that induce desired structural modifications. By integrating Bayesian inference, the fluctuation–dissipation theorem, and geometric analysis of the loss landscape, this study unifies the theoretical formulations of influence functions and structural susceptibility, provides a computable framework for evaluating sensitivity, and offers an efficient linear approach to controlling neural network behavior.

0 citationsRead paper

Patterning: The Dual of Interpretability

Jan 20, 2026

This work proposes a “patterning” approach that reframes interpretability as the active shaping of a neural network’s internal structure and generalization behavior through deliberate training data design. Grounded in linear response theory, the method employs susceptibilities to quantify model sensitivity to data perturbations and inversely solves for optimal data intervention strategies—such as reweighting and localized learning coefficient optimization—to directionally control the formation of inductive circuits. Experiments on small language models demonstrate that this technique can effectively accelerate or delay the emergence of specific circuits and, in the context of a bracket balancing task, guide the model to learn a prescribed algorithm. This represents the first demonstration of actively writing and regulating internal model structures through targeted data interventions.

0 citationsRead paper

Towards Spectroscopy: Susceptibility Clusters in Language Models

Jan 19, 2026

This study addresses the need to uncover the internal structure of language models and their response mechanisms to input perturbations. To this end, it introduces spectroscopic principles into language model analysis and proposes a susceptibility-based clustering method. By perturbing the distribution of context tokens and employing Stochastic Gradient Langevin Dynamics (SGLD) to approximate the local Gibbs posterior, the approach combines admittance clustering with covariance analysis to identify semantic units organized by similar causal mechanisms within the model. Applied to Pythia-14M, the method successfully discovers 510 interpretable clusters capturing patterns in syntax, code, and mathematical notation, with 50% showing correspondence to features extracted by sparse autoencoders, thereby validating both the efficacy and interpretability of the proposed framework.

0 citationsRead paper

Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

Oct 13, 2025

This study addresses the theoretical assessment of neural network compressibility limits. We extend the Minimum Description Length (MDL) principle—traditionally applicable only to regular models—to singular statistical models by integrating singular learning theory, and propose a novel model complexity estimator based on the Local Learning Coefficient (LLC). Systematic evaluation on the Pythia model family demonstrates a strong linear correlation between LLC and achievable compression ratios across diverse techniques, including quantization and tensor decomposition. Our approach yields the first computationally tractable, theoretically grounded complexity measure for neural networks, overcoming the fundamental limitation that conventional MDL is inapplicable to non-regular (singular) models. By establishing a principled, interpretable link between intrinsic model complexity and compressibility, this work provides a rigorous theoretical framework for characterizing fundamental compression limits in deep learning.

0 citationsRead paper