Selected work

Representative Papers

Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA

Jun 30, 2026

This work addresses the vulnerability of self-generated question-answering (QA) as a training signal for language models, which is prone to selection bias and excessive compliance with embedded instructions. The study systematically reveals the model’s preferential selection of document evidence and its high adherence to implicit directives during self-generated QA. To mitigate these issues without altering the training pipeline, the authors propose lightweight interventions: fixing the questioning objective and filtering instruction-laden segments. Through comprehensive empirical evaluations—including multi-model comparisons, prompt diversity analysis, text coverage assessment, and instruction injection detection—the approach reduces average instruction compliance from 88% to 13% while preserving nearly all clean textual content. This significantly enhances the reliability and robustness of self-generated QA data for model training.

0 citationsRead paper

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Jun 02, 2026

This study addresses the lack of systematic evaluation of tensor decomposition methods for post-training compression of large language models (LLMs), particularly regarding their applicability to both dense and mixture-of-experts (MoE) architectures. Through a combination of theoretical analysis and empirical experiments, this work provides the first comprehensive assessment of tensor decomposition across diverse LLM architectures, revealing critical performance trade-offs. It identifies a fundamental mismatch between the shared subspace assumption inherent in tensor decomposition and the heterogeneous representations actually learned by modern LLMs. By delineating the practical boundaries and limitations of tensor-based compression in contemporary LLMs, this research offers actionable insights for efficient model deployment and releases its implementation code to facilitate further investigation.

0 citationsRead paper

Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery

Jun 01, 2026

This work addresses the instability and repetitive generation pathologies that commonly arise in large language model inference under extreme low-bit quantization (e.g., 2-bit), which often nullify end-to-end acceleration benefits. The study is the first to attribute such failures to controllable generation pathologies and introduces two lightweight intervention mechanisms: FP16 high-precision planning and repetition-trajectory-aware selective fallback. Experiments on the Qwen3 series demonstrate that these strategies dramatically restore reasoning accuracy while preserving computational efficiency—boosting the 2-bit Qwen3-8B’s performance on MATH-500 from 17.2% to 74.2%, and achieving 87.2% with Qwen3-32B when both mechanisms are combined.

0 citationsRead paper

HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization

May 28, 2026

Extreme low-bit (2–4 bit) quantization of large language models suffers significant performance degradation due to activation outliers and anisotropic weight curvature. To address this, this work proposes HARP, a learnable structured bilateral orthogonal processor that replaces the fixed Hadamard transform with an adaptive quantization basis while preserving full-precision equivalence. HARP introduces, for the first time, an adaptive rotation mechanism based on Hadamard preconditioning, supporting arbitrary dimensions and enabling dynamic adaptation across layers, calibration data, and quantizers. Its rotation matrix is efficiently parameterized via sparse butterfly-block orthogonal products, combined with mixed-radix scheduling and Hadamard initialization, allowing effective fitting using only calibration data. Experiments demonstrate that HARP substantially improves perplexity and zero-shot accuracy across models ranging from 1B to 70B parameters, achieving an inference speed of 128 tokens per second—more than double the 61 tokens per second of FP16.

0 citationsRead paper

Hierarchical Mixture-of-Experts with Two-Stage Optimization

May 08, 2026

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

0 citationsRead paper

Recent publications

Latest Papers

Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA

Jun 30, 2026

This work addresses the vulnerability of self-generated question-answering (QA) as a training signal for language models, which is prone to selection bias and excessive compliance with embedded instructions. The study systematically reveals the model’s preferential selection of document evidence and its high adherence to implicit directives during self-generated QA. To mitigate these issues without altering the training pipeline, the authors propose lightweight interventions: fixing the questioning objective and filtering instruction-laden segments. Through comprehensive empirical evaluations—including multi-model comparisons, prompt diversity analysis, text coverage assessment, and instruction injection detection—the approach reduces average instruction compliance from 88% to 13% while preserving nearly all clean textual content. This significantly enhances the reliability and robustness of self-generated QA data for model training.

0 citationsRead paper

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Jun 02, 2026

This study addresses the lack of systematic evaluation of tensor decomposition methods for post-training compression of large language models (LLMs), particularly regarding their applicability to both dense and mixture-of-experts (MoE) architectures. Through a combination of theoretical analysis and empirical experiments, this work provides the first comprehensive assessment of tensor decomposition across diverse LLM architectures, revealing critical performance trade-offs. It identifies a fundamental mismatch between the shared subspace assumption inherent in tensor decomposition and the heterogeneous representations actually learned by modern LLMs. By delineating the practical boundaries and limitations of tensor-based compression in contemporary LLMs, this research offers actionable insights for efficient model deployment and releases its implementation code to facilitate further investigation.

0 citationsRead paper

Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery

Jun 01, 2026

This work addresses the instability and repetitive generation pathologies that commonly arise in large language model inference under extreme low-bit quantization (e.g., 2-bit), which often nullify end-to-end acceleration benefits. The study is the first to attribute such failures to controllable generation pathologies and introduces two lightweight intervention mechanisms: FP16 high-precision planning and repetition-trajectory-aware selective fallback. Experiments on the Qwen3 series demonstrate that these strategies dramatically restore reasoning accuracy while preserving computational efficiency—boosting the 2-bit Qwen3-8B’s performance on MATH-500 from 17.2% to 74.2%, and achieving 87.2% with Qwen3-32B when both mechanisms are combined.

0 citationsRead paper

HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization

May 28, 2026

Extreme low-bit (2–4 bit) quantization of large language models suffers significant performance degradation due to activation outliers and anisotropic weight curvature. To address this, this work proposes HARP, a learnable structured bilateral orthogonal processor that replaces the fixed Hadamard transform with an adaptive quantization basis while preserving full-precision equivalence. HARP introduces, for the first time, an adaptive rotation mechanism based on Hadamard preconditioning, supporting arbitrary dimensions and enabling dynamic adaptation across layers, calibration data, and quantizers. Its rotation matrix is efficiently parameterized via sparse butterfly-block orthogonal products, combined with mixed-radix scheduling and Hadamard initialization, allowing effective fitting using only calibration data. Experiments demonstrate that HARP substantially improves perplexity and zero-shot accuracy across models ranging from 1B to 70B parameters, achieving an inference speed of 128 tokens per second—more than double the 61 tokens per second of FP16.

0 citationsRead paper

Hierarchical Mixture-of-Experts with Two-Stage Optimization

May 08, 2026

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

0 citationsRead paper