LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

2026-01-28

AI Summary

This work addresses the challenge of deploying large multimodal models, which suffer from high computational and memory costs, while existing compression techniques often incur significant reconstruction error due to the decoupled application of low-rank decomposition and quantization. To overcome this limitation, the authors propose a frequency-domain joint compression framework that simultaneously performs low-rank approximation and quantization in the Fourier domain, leveraging its decorrelation and conjugate symmetry properties to yield more compact and accurate weight representations. Key innovations include PolarQuant, a polar-coordinate-based quantization method tailored for complex-valued matrices, and an optional diagonal calibration (ODC) scheme that operates without extensive calibration data. Experiments demonstrate that the proposed approach substantially reduces activation memory and computational overhead across multiple multimodal benchmarks, while maintaining or even surpassing the performance of current efficient models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.

Problem

• large multimodal models
• model compression
• low-rank decomposition
• quantization
• cross-modal redundancy

Innovation

• Fourier approximation
• joint low-rank quantization
• PolarQuant
• frequency-domain compression
• multimodal model efficiency