Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs

2026-08-21

AI Summary

为解决视觉语言模型在移动设备上的部署问题，提出了一种结合自生成训练数据的量化管道和2.7位参数格式的高效量化框架。

Abstract

Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.

Problem

• vision-language models
• mobile devices
• memory and compute requirements

Innovation

• Efficient Quantization
• Resource-Constrained Hardware
• 2.7-Bit-Per-Parameter
• Vision-Language Models
• Mobile Deployment