Selected work

Representative Papers

Layer-wise Quantization for Quantized Optimistic Dual Averaging

May 20, 2025

The inter-layer heterogeneity of deep neural networks—e.g., residual blocks and multi-head attention modules—exhibits significant disparities in dimensionality, activation patterns, and representation characteristics, leading to excessive communication overhead and slow convergence in distributed variational inequality (VI) optimization. Method: This paper introduces layer-aware quantization into the VI optimization framework for the first time, proposing a layer-adaptive quantization mechanism and the Quantized Optimistic Dual Averaging (QODA) algorithm. Contribution/Results: We derive tight bounds on quantization variance and minimum code length; design an adaptive step-size strategy ensuring optimal $O(1/T)$ convergence under monotone VIs. Evaluated on a 12+ GPU cluster training Wasserstein GANs, our method achieves a 150% end-to-end speedup, substantially outperforming existing quantization-based and distributed VI approaches.

0 citationsRead paper

HALO: Hadamard-Assisted Lossless Optimization for Efficient Low-Precision LLM Training and Fine-Tuning

Jan 05, 2025

Low-precision full quantization fine-tuning of large language models (LLMs) suffers from severe accuracy degradation due to weight/activation outliers. Method: This work introduces the first practical FP8 full-quantization fine-tuning framework. It proposes a novel Hadamard rotation preprocessing to suppress outliers, integrated with quantization-aware training (QAT), FSDP-based low-precision communication optimization, and custom FP8 CUDA kernels—enabling stable FP8 computation across all large matrix multiplications in both forward and backward passes. The framework is compatible with standard fine-tuning and parameter-efficient fine-tuning (PEFT) without requiring additional hyperparameter tuning. Contribution/Results: On LLaMA-family models, it achieves lossless FP8 fine-tuning accuracy while delivering 1.31× end-to-end speedup on RTX 4090. This work provides the first empirical validation of feasibility, stability, and efficiency of full-quantization LLM fine-tuning.

0 citationsRead paper

Recent publications

Latest Papers

Layer-wise Quantization for Quantized Optimistic Dual Averaging

May 20, 2025

The inter-layer heterogeneity of deep neural networks—e.g., residual blocks and multi-head attention modules—exhibits significant disparities in dimensionality, activation patterns, and representation characteristics, leading to excessive communication overhead and slow convergence in distributed variational inequality (VI) optimization. Method: This paper introduces layer-aware quantization into the VI optimization framework for the first time, proposing a layer-adaptive quantization mechanism and the Quantized Optimistic Dual Averaging (QODA) algorithm. Contribution/Results: We derive tight bounds on quantization variance and minimum code length; design an adaptive step-size strategy ensuring optimal $O(1/T)$ convergence under monotone VIs. Evaluated on a 12+ GPU cluster training Wasserstein GANs, our method achieves a 150% end-to-end speedup, substantially outperforming existing quantization-based and distributed VI approaches.

0 citationsRead paper

HALO: Hadamard-Assisted Lossless Optimization for Efficient Low-Precision LLM Training and Fine-Tuning

Jan 05, 2025

Low-precision full quantization fine-tuning of large language models (LLMs) suffers from severe accuracy degradation due to weight/activation outliers. Method: This work introduces the first practical FP8 full-quantization fine-tuning framework. It proposes a novel Hadamard rotation preprocessing to suppress outliers, integrated with quantization-aware training (QAT), FSDP-based low-precision communication optimization, and custom FP8 CUDA kernels—enabling stable FP8 computation across all large matrix multiplications in both forward and backward passes. The framework is compatible with standard fine-tuning and parameter-efficient fine-tuning (PEFT) without requiring additional hyperparameter tuning. Contribution/Results: On LLaMA-family models, it achieves lossless FP8 fine-tuning accuracy while delivering 1.31× end-to-end speedup on RTX 4090. This work provides the first empirical validation of feasibility, stability, and efficiency of full-quantization LLM fine-tuning.

0 citationsRead paper