Selected work

Representative Papers

Visual Instruction Tuning Aligns Modalities through Abstraction

Jun 02, 2026

This work investigates how visual instruction tuning integrates image information into the hierarchical architecture of large language models—a mechanism not well understood in prior studies. Through a systematic analysis employing probing, causal intervention, representational geometry comparison, and selective layer fine-tuning, the study reveals that visual features are predominantly embedded in the model’s intermediate semantic layers in a localized manner. Building on this insight, the authors demonstrate that fine-tuning only these intermediate layers achieves performance comparable to full-model fine-tuning across multiple vision-centric benchmarks, while substantially reducing training costs. These findings underscore the pivotal role of intermediate layers in cross-modal alignment and offer an efficient strategy for multimodal adaptation.

0 citationsRead paper

Density-Informed VAE (DiVAE): Reliable Log-Prior Probability via Density Alignment Regularization

Dec 03, 2025

Standard VAEs enforce latent variables to match simplistic priors (e.g., standard normal), neglecting the true data density structure. This causes misalignment between the log-prior and log-data-density, degrading distribution alignment, prior coverage, out-of-distribution (OOD) detection, and uncertainty calibration. To address this, we propose DiVAE: a variational autoencoder incorporating a lightweight, data-driven density alignment regularizer within the ELBO framework. This term explicitly aligns the log-prior in latent space with the log-density of input data, estimated via nonparametric or parametric density estimation. Concurrently, it encourages the learnable prior to concentrate around high-density data regions and adaptively allocates posterior mass according to local data density. Experiments on synthetic datasets and MNIST demonstrate that DiVAE significantly improves latent-space density consistency, prior coverage, and OOD detection performance—enhancing both model interpretability and reliability.

0 citationsRead paper

Interpreting and Steering Protein Language Models through Sparse Autoencoders

Feb 13, 2025

Understanding the internal representational mechanisms of protein language models (PLMs) remains challenging. Method: We introduce the first sparse autoencoder (SAE)-based interpretability framework tailored for PLMs—specifically ESM-2 (8M)—that maps hidden-layer activations to biologically annotated functional motifs (e.g., transmembrane regions, binding sites, zinc finger domains). Statistical significance of function-specific neurons is assessed via Fisher’s exact test. Furthermore, we develop a conditional generation strategy guided by activations of identified interpretable neurons to enable targeted sequence design of target motifs (e.g., signal peptides, zinc fingers). Contribution/Results: Our approach successfully identifies multiple biologically grounded, interpretable neurons and achieves over threefold enrichment of target motifs in generated sequences. This work establishes a novel paradigm for mechanistic understanding and controllable design using PLMs.

0 citationsRead paper

Recent publications

Latest Papers

Visual Instruction Tuning Aligns Modalities through Abstraction

Jun 02, 2026

This work investigates how visual instruction tuning integrates image information into the hierarchical architecture of large language models—a mechanism not well understood in prior studies. Through a systematic analysis employing probing, causal intervention, representational geometry comparison, and selective layer fine-tuning, the study reveals that visual features are predominantly embedded in the model’s intermediate semantic layers in a localized manner. Building on this insight, the authors demonstrate that fine-tuning only these intermediate layers achieves performance comparable to full-model fine-tuning across multiple vision-centric benchmarks, while substantially reducing training costs. These findings underscore the pivotal role of intermediate layers in cross-modal alignment and offer an efficient strategy for multimodal adaptation.

0 citationsRead paper

Density-Informed VAE (DiVAE): Reliable Log-Prior Probability via Density Alignment Regularization

Dec 03, 2025

Standard VAEs enforce latent variables to match simplistic priors (e.g., standard normal), neglecting the true data density structure. This causes misalignment between the log-prior and log-data-density, degrading distribution alignment, prior coverage, out-of-distribution (OOD) detection, and uncertainty calibration. To address this, we propose DiVAE: a variational autoencoder incorporating a lightweight, data-driven density alignment regularizer within the ELBO framework. This term explicitly aligns the log-prior in latent space with the log-density of input data, estimated via nonparametric or parametric density estimation. Concurrently, it encourages the learnable prior to concentrate around high-density data regions and adaptively allocates posterior mass according to local data density. Experiments on synthetic datasets and MNIST demonstrate that DiVAE significantly improves latent-space density consistency, prior coverage, and OOD detection performance—enhancing both model interpretability and reliability.

0 citationsRead paper

Interpreting and Steering Protein Language Models through Sparse Autoencoders

Feb 13, 2025

Understanding the internal representational mechanisms of protein language models (PLMs) remains challenging. Method: We introduce the first sparse autoencoder (SAE)-based interpretability framework tailored for PLMs—specifically ESM-2 (8M)—that maps hidden-layer activations to biologically annotated functional motifs (e.g., transmembrane regions, binding sites, zinc finger domains). Statistical significance of function-specific neurons is assessed via Fisher’s exact test. Furthermore, we develop a conditional generation strategy guided by activations of identified interpretable neurons to enable targeted sequence design of target motifs (e.g., signal peptides, zinc fingers). Contribution/Results: Our approach successfully identifies multiple biologically grounded, interpretable neurons and achieves over threefold enrichment of target motifs in generated sequences. This work establishes a novel paradigm for mechanistic understanding and controllable design using PLMs.

0 citationsRead paper