Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

2026-07-08

AI Summary

This study addresses the lack of systematic understanding of component-wise quantization behavior in small-to-medium-scale vision-language models deployed at the edge, which hinders efficient and low-power implementations. Conducted on Jetson Orin NX and AGX platforms, the work performs component-level quantization evaluation across visual encoders, projection modules, and LLM backbones, validating five key hypotheses. Through six INT4/INT8 quantization configurations and comprehensive measurements of latency, VRAM usage, and energy efficiency, the study reveals that quantization sensitivity is primarily governed by architectural choices—such as Mixture-of-Experts outperforming dense models—rather than parameter count alone. It further identifies encoder-kernel-hardware coupling as the root cause of SigLIP latency, and demonstrates the platform-dependent nature of non-additive quantization errors and energy efficiency along the modality alignment pathway. These findings provide both theoretical grounding and practical guidance for quantized edge deployment.

Abstract

The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.

Problem

• vision language models
• quantization
• edge deployment
• hardware-aware
• component-wise analysis

Innovation

• component-wise quantization
• hardware-aware deployment
• vision-language models
• MoE vs. dense
• intelligence-per-joule