Selected work

Representative Papers

Rethinking the Rank Threshold for LoRA Fine-Tuning

May 05, 2026

This work addresses the limitations of existing LoRA fine-tuning theory, which imposes overly stringent rank requirements and lacks adaptation to cross-entropy loss in binary classification settings. By refining manifold dimension analysis and incorporating the Polyak–Łojasiewicz inequality with Rademacher complexity bounds, the authors establish, for the first time within the neural tangent kernel (NTK) framework, optimization guarantees for binary classification tasks that require either no rank threshold or an extremely low rank—such as rank 1. Theoretical analysis demonstrates that the necessary LoRA rank can be reduced from the commonly recommended value of 12 down to 1. Empirical validation across four GLUE binary classification benchmarks and multiple encoder architectures confirms that rank-1 LoRA achieves performance comparable to rank-12, while multi-class tasks exhibit higher optimal ranks, aligning with theoretical predictions.

0 citationsRead paper

Decoder-Free Distillation for Quantized Image Restoration

Mar 10, 2026

This work addresses key limitations of quantization-aware training and knowledge distillation in image restoration, including teacher–student capacity mismatch, spatial error amplification caused by decoder-side distillation, and conflict between reconstruction and distillation losses due to quantization noise. To overcome these challenges, the authors propose the QDR framework, which eliminates capacity gaps via FP32 self-distillation, introduces decoder-free distillation (DFD) at the bottleneck to correct quantization errors, and employs learnable magnitude reweighting (LMR) to dynamically balance gradient conflicts. Additionally, a learnable degradation gating (LDG) module is incorporated to enhance robustness. The proposed method recovers 96.5% of FP32 performance under INT8 quantization, achieves 442 FPS on an NVIDIA Jetson Orin, and improves downstream object detection by 16.3 mAP.

0 citationsRead paper

Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning

Feb 13, 2026

On-device fine-tuning enables privacy-preserving personalization of large language models, but mobile devices impose severe memory constraints, typically 6--12GB shared across all workloads. Existing approaches force a trade-off between exact gradients with high memory (MeBP) and low memory with noisy estimates (MeZO). We propose Memory-efficient Structured Backpropagation (MeSP), which bridges this gap by manually deriving backward passes that exploit LoRA's low-rank structure. Our key insight is that the intermediate projection $h = xA$ can be recomputed during backward at minimal cost since rank $r \ll d_{in}$, eliminating the need to store it. MeSP achieves 49\% average memory reduction compared to MeBP on Qwen2.5 models (0.5B--3B) while computing mathematically identical gradients. Our analysis also reveals that MeZO's gradient estimates show near-zero correlation with true gradients (cosine similarity $\approx$0.001), explaining its slow convergence. MeSP reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices.

0 citationsRead paper

LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning

Feb 13, 2026

Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large language models (LLMs) on mobile devices with less than 1GB memory. However, MeBP requires backward computation through all transformer layers at every step, where weight decompression alone accounts for 32--42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40$\times$ speedup with less than 2\% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.

0 citationsRead paper

Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer

Sep 25, 2025

In low-bit quantization-aware training (QAT) combined with knowledge distillation (KD), optimization conflicts arise due to heterogeneous gradient magnitudes between task loss and distillation loss. Method: We propose the Game of Regularizer (GoR), a lightweight dynamic regularization mechanism with only two learnable parameters, which adaptively balances supervision signals to mitigate gradient conflict. Building upon GoR, we introduce QAT-EKD-GoR—a unified framework supporting ensemble KD from multiple teachers. Contribution/Results: QAT-EKD-GoR achieves state-of-the-art performance across image classification, object detection, and large language model compression. In several cases, it even surpasses full-precision baselines in accuracy while significantly improving inference efficiency on edge devices—effectively reconciling high accuracy with ultra-low power consumption.

0 citationsRead paper

Recent publications

Latest Papers

Rethinking the Rank Threshold for LoRA Fine-Tuning

May 05, 2026

This work addresses the limitations of existing LoRA fine-tuning theory, which imposes overly stringent rank requirements and lacks adaptation to cross-entropy loss in binary classification settings. By refining manifold dimension analysis and incorporating the Polyak–Łojasiewicz inequality with Rademacher complexity bounds, the authors establish, for the first time within the neural tangent kernel (NTK) framework, optimization guarantees for binary classification tasks that require either no rank threshold or an extremely low rank—such as rank 1. Theoretical analysis demonstrates that the necessary LoRA rank can be reduced from the commonly recommended value of 12 down to 1. Empirical validation across four GLUE binary classification benchmarks and multiple encoder architectures confirms that rank-1 LoRA achieves performance comparable to rank-12, while multi-class tasks exhibit higher optimal ranks, aligning with theoretical predictions.

0 citationsRead paper

Decoder-Free Distillation for Quantized Image Restoration

Mar 10, 2026

This work addresses key limitations of quantization-aware training and knowledge distillation in image restoration, including teacher–student capacity mismatch, spatial error amplification caused by decoder-side distillation, and conflict between reconstruction and distillation losses due to quantization noise. To overcome these challenges, the authors propose the QDR framework, which eliminates capacity gaps via FP32 self-distillation, introduces decoder-free distillation (DFD) at the bottleneck to correct quantization errors, and employs learnable magnitude reweighting (LMR) to dynamically balance gradient conflicts. Additionally, a learnable degradation gating (LDG) module is incorporated to enhance robustness. The proposed method recovers 96.5% of FP32 performance under INT8 quantization, achieves 442 FPS on an NVIDIA Jetson Orin, and improves downstream object detection by 16.3 mAP.

0 citationsRead paper

Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning

Feb 13, 2026

On-device fine-tuning enables privacy-preserving personalization of large language models, but mobile devices impose severe memory constraints, typically 6--12GB shared across all workloads. Existing approaches force a trade-off between exact gradients with high memory (MeBP) and low memory with noisy estimates (MeZO). We propose Memory-efficient Structured Backpropagation (MeSP), which bridges this gap by manually deriving backward passes that exploit LoRA's low-rank structure. Our key insight is that the intermediate projection $h = xA$ can be recomputed during backward at minimal cost since rank $r \ll d_{in}$, eliminating the need to store it. MeSP achieves 49\% average memory reduction compared to MeBP on Qwen2.5 models (0.5B--3B) while computing mathematically identical gradients. Our analysis also reveals that MeZO's gradient estimates show near-zero correlation with true gradients (cosine similarity $\approx$0.001), explaining its slow convergence. MeSP reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices.

0 citationsRead paper

LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning

Feb 13, 2026

Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large language models (LLMs) on mobile devices with less than 1GB memory. However, MeBP requires backward computation through all transformer layers at every step, where weight decompression alone accounts for 32--42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40$\times$ speedup with less than 2\% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.

0 citationsRead paper

Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer

Sep 25, 2025

In low-bit quantization-aware training (QAT) combined with knowledge distillation (KD), optimization conflicts arise due to heterogeneous gradient magnitudes between task loss and distillation loss. Method: We propose the Game of Regularizer (GoR), a lightweight dynamic regularization mechanism with only two learnable parameters, which adaptively balances supervision signals to mitigate gradient conflict. Building upon GoR, we introduce QAT-EKD-GoR—a unified framework supporting ensemble KD from multiple teachers. Contribution/Results: QAT-EKD-GoR achieves state-of-the-art performance across image classification, object detection, and large language model compression. In several cases, it even surpasses full-precision baselines in accuracy while significantly improving inference efficiency on edge devices—effectively reconciling high accuracy with ultra-low power consumption.

0 citationsRead paper