E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring

2026-05-16

AI Summary

This work addresses the significant performance degradation in post-merge quantization (PMQ), which arises from the coupling of quantization bias with the relative expert bias introduced during model merging. The study formally defines the PMQ problem for the first time and introduces the E-PMQ framework, which effectively decouples these dual biases through expert-guided layer-wise calibration and a merged-weight anchoring mechanism. This approach stabilizes the calibration process while preserving ensemble behavior. Combining 4-bit GPTQ with fusion strategies such as Task Arithmetic and TIES-Merging, E-PMQ achieves 73.6%–74.8% accuracy across eight tasks on CLIP-ViT-B/32 and dramatically improves performance on CLIP-ViT-L/14 from 34.8% to 76.7% across twenty tasks. Furthermore, it attains 83.34% on the FLAN-T5-base GLUE benchmark.

Abstract

Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increasingly practical low-resource strategy for integrating multiple task- or domain-specialized experts into a single model without joint training or multi-model serving. Together, quantization and model merging enable an efficient low-resource deployment pipeline by integrating multiple experts into one low-bit model. We formulate this setting as Post-Merge Quantization (PMQ). We show that directly applying post-training quantization (PTQ) to a merged model is unreliable because two distinct deviations are coupled: the quantization deviation introduced by low-bit reconstruction and the expert-relative merging deviation inherited from model merging. To mitigate these deviations, we propose E-PMQ, an expert-guided PMQ framework that uses source expert weights to provide expert- guided output targets during layer-wise calibration, together with merged-weight anchoring to stabilize the calibration and preserve the integrated behavior of the merged model. On CLIP-ViT-B/32 eight-task merging, E-PMQ improves 4-bit GPTQ from 65.0% to 73.6% under Task Arithmetic and from 69.1% to 74.8% under TIES-Merging. On harder settings, E-PMQ improves GPTQ from 34.8% to 76.7% on 20-task CLIP-ViT-L/14 and from 78.26% to 83.34% on FLAN-T5- base GLUE. These results demonstrate that E-PMQ enables effective post-merge quantization and low-bit deployment.

Problem

• Post-Merge Quantization
• Model Merging
• Quantization Deviation
• Low-Resource Deployment
• Expert Models

Innovation

• Post-Merge Quantization
• Expert-Guided Calibration
• Merged-Weight Anchoring
• Model Merging
• Low-Bit Deployment