Selected work

Representative Papers

Decomposing the Depth Profile of Fine-Tuning

Apr 18, 2026

This study investigates whether the depth-wise distribution of representational changes during fine-tuning arises from intrinsic model properties or the magnitude of gradient flow. Through 240 fine-tuning experiments spanning serial and parallel architectures and model scales from 125M to 6.9B parameters, combined with representational similarity analysis, layer-wise relative weight change (|ΔW|/|W|), and task-agnostic target distance metrics, the work systematically reveals that fine-tuning depth profiles are jointly governed by architecture type, task objective, and model scale. Under standard training, representational shifts concentrate in upper layers; under controlled conditions, only serial architectures retain a positive slope in small models, while parallel architectures maintain it solely in causal language modeling. Above 1.3B parameters, both architectures exhibit positive slopes, with profile steepness correlating with initial target distance and width primarily dictated by architecture—demonstrating that “localized gradients” emerge from a multifactorial interplay.

0 citationsRead paper

Predicting Where Steering Vectors Succeed

Apr 16, 2026

Existing methods struggle to predict where semantic vector interventions will be effective across concepts and model layers. This work proposes the Linear Accessibility Profile (LAP), a training-free layer diagnostic metric, \(A_{\text{lin}}\), that integrates the logit lens, unembedding matrices, and mean-difference analysis to accurately forecast both intervention efficacy and the optimal layer for intervention—achieving this capability for the first time. Grounded in a three-mechanism framework, LAP delineates the conditions under which linear semantic vectors succeed, require nonlinear alternatives, or fail entirely. Experiments across five models and 24 binary concept pairs show that \(A_{\text{lin}}\) correlates strongly (0.86–0.91) with actual intervention outcomes, and that layers selected by LAP significantly outperform default middle layers, yielding end-to-end improvements validated on Gemma-2-2B and OLMo-2-1B-Instruct.

0 citationsRead paper

Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs

Feb 26, 2026

This work addresses the “modality collapse” phenomenon in multimodal large language models, wherein decoders trained solely toward textual alignment fail to effectively leverage modality-specific cues—such as speaker identity and emotion in speech or visual textures in images. The study formalizes this issue through an information-theoretic lens as a mismatch between the decoder and the input distribution, and introduces Generalized Mutual Information (GMI) as a theoretical upper bound on accessible information. Through linear probing, LoRA fine-tuning, and controlled contrastive experiments across five audio-visual models, the authors demonstrate that optimizing the decoding objective significantly enhances the extractability of non-textual information: LoRA-based intervention boosts emotion information accessibility by 7.5% without degrading other attributes, thereby establishing that the training objective fundamentally governs information usability.

0 citationsRead paper

The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

Feb 19, 2026

This study investigates whether speech large language models (Speech LLMs) are behaviorally and mechanistically equivalent to cascaded systems comprising an automatic speech recognition (ASR) model followed by a large language model (LLM). By aligning the LLM backbone architectures, the authors systematically compare four Speech LLMs against a Whisper→LLM cascade across six tasks and propose the “cascade equivalence hypothesis.” Leveraging logit lens analysis, LEACE-based concept erasure, noise robustness evaluations, and causal interventions, they provide the first evidence for the necessity of textual representations in such models. Results show that Ultravox closely mirrors its cascaded counterpart (κ = 0.93), with performance collapsing upon textual representation erasure, while Qwen2-Audio exhibits significant deviation. Moreover, most Speech LLMs outperform their cascaded equivalents by up to 7.6% under noisy conditions. The findings demonstrate that cascade equivalence is architecture-dependent and not universally valid.

0 citationsRead paper

Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks

Feb 17, 2026

This study investigates the intrinsic mechanisms underlying emergent capabilities in neural networks, with a focus on how the geometric structure of representations evolves with model scale. By analyzing five geometric metrics—including RANKME, local learning coefficients, and Hessian-based measures—across over 120 emergence events in Pythia models ranging from 405M to 2.8B parameters, the work reveals that representational collapse exhibits scale-invariant properties and propagates top-down through the network. The findings demonstrate that a hierarchical geometric structure governs capability emergence, and that these geometric indicators can predict the emergence of difficult tasks up to 75–100% in advance under task-aligned conditions, achieving consistent results across all 32 tasks and models tested. However, this predictive power vanishes in unsupervised pretraining due to the absence of task alignment.

0 citationsRead paper

Recent publications

Latest Papers

Decomposing the Depth Profile of Fine-Tuning

Apr 18, 2026

This study investigates whether the depth-wise distribution of representational changes during fine-tuning arises from intrinsic model properties or the magnitude of gradient flow. Through 240 fine-tuning experiments spanning serial and parallel architectures and model scales from 125M to 6.9B parameters, combined with representational similarity analysis, layer-wise relative weight change (|ΔW|/|W|), and task-agnostic target distance metrics, the work systematically reveals that fine-tuning depth profiles are jointly governed by architecture type, task objective, and model scale. Under standard training, representational shifts concentrate in upper layers; under controlled conditions, only serial architectures retain a positive slope in small models, while parallel architectures maintain it solely in causal language modeling. Above 1.3B parameters, both architectures exhibit positive slopes, with profile steepness correlating with initial target distance and width primarily dictated by architecture—demonstrating that “localized gradients” emerge from a multifactorial interplay.

0 citationsRead paper

Predicting Where Steering Vectors Succeed

Apr 16, 2026

Existing methods struggle to predict where semantic vector interventions will be effective across concepts and model layers. This work proposes the Linear Accessibility Profile (LAP), a training-free layer diagnostic metric, \(A_{\text{lin}}\), that integrates the logit lens, unembedding matrices, and mean-difference analysis to accurately forecast both intervention efficacy and the optimal layer for intervention—achieving this capability for the first time. Grounded in a three-mechanism framework, LAP delineates the conditions under which linear semantic vectors succeed, require nonlinear alternatives, or fail entirely. Experiments across five models and 24 binary concept pairs show that \(A_{\text{lin}}\) correlates strongly (0.86–0.91) with actual intervention outcomes, and that layers selected by LAP significantly outperform default middle layers, yielding end-to-end improvements validated on Gemma-2-2B and OLMo-2-1B-Instruct.

0 citationsRead paper

Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs

Feb 26, 2026

This work addresses the “modality collapse” phenomenon in multimodal large language models, wherein decoders trained solely toward textual alignment fail to effectively leverage modality-specific cues—such as speaker identity and emotion in speech or visual textures in images. The study formalizes this issue through an information-theoretic lens as a mismatch between the decoder and the input distribution, and introduces Generalized Mutual Information (GMI) as a theoretical upper bound on accessible information. Through linear probing, LoRA fine-tuning, and controlled contrastive experiments across five audio-visual models, the authors demonstrate that optimizing the decoding objective significantly enhances the extractability of non-textual information: LoRA-based intervention boosts emotion information accessibility by 7.5% without degrading other attributes, thereby establishing that the training objective fundamentally governs information usability.

0 citationsRead paper

The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

Feb 19, 2026

This study investigates whether speech large language models (Speech LLMs) are behaviorally and mechanistically equivalent to cascaded systems comprising an automatic speech recognition (ASR) model followed by a large language model (LLM). By aligning the LLM backbone architectures, the authors systematically compare four Speech LLMs against a Whisper→LLM cascade across six tasks and propose the “cascade equivalence hypothesis.” Leveraging logit lens analysis, LEACE-based concept erasure, noise robustness evaluations, and causal interventions, they provide the first evidence for the necessity of textual representations in such models. Results show that Ultravox closely mirrors its cascaded counterpart (κ = 0.93), with performance collapsing upon textual representation erasure, while Qwen2-Audio exhibits significant deviation. Moreover, most Speech LLMs outperform their cascaded equivalents by up to 7.6% under noisy conditions. The findings demonstrate that cascade equivalence is architecture-dependent and not universally valid.

0 citationsRead paper

Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks

Feb 17, 2026

This study investigates the intrinsic mechanisms underlying emergent capabilities in neural networks, with a focus on how the geometric structure of representations evolves with model scale. By analyzing five geometric metrics—including RANKME, local learning coefficients, and Hessian-based measures—across over 120 emergence events in Pythia models ranging from 405M to 2.8B parameters, the work reveals that representational collapse exhibits scale-invariant properties and propagates top-down through the network. The findings demonstrate that a hierarchical geometric structure governs capability emergence, and that these geometric indicators can predict the emergence of difficult tasks up to 75–100% in advance under task-aligned conditions, achieving consistent results across all 32 tasks and models tested. However, this predictive power vanishes in unsupervised pretraining due to the absence of task alignment.

0 citationsRead paper