Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration

2026-06-15

AI Summary

This work addresses the modality imbalance in unified multimodal models during instruction tuning, where language gradients often dominate and severely degrade image generation quality—a problem particularly pronounced in parameter-efficient fine-tuning methods like LoRA. The study presents the first systematic analysis of gradient disparities in multimodal LoRA fine-tuning and formulates the issue as a bi-objective optimization problem. To resolve it, the authors propose Pareto LoRA, a novel approach that dynamically balances the direction and magnitude of gradients from text and image objectives through a Pareto-optimal gradient integration mechanism. Experiments on the CoMM benchmark using the Emu2 model demonstrate that Pareto LoRA preserves strong text generation performance while improving image perceptual quality by up to 44.9%.

Abstract

Unified multimodal models (UMMs) have recently emerged as a promising paradigm for integrating multimodal understanding and generation within a single autoregressive transformer. However, during multimodal instruction tuning, these models often exhibit pronounced modality imbalance: language gradients dominate optimization, thus leading to lower image generation quality, especially under parameter-efficient fine-tuning such as LoRA. In this work, we systematically analyze modality imbalance in LoRA-based fine-tuning of UMMs for interleaved text-image generation. We show that vision modality performance degrades substantially more than text modality performance when compared to unimodal counterparts, and that modality-specific gradients can differ by orders of magnitude across various tasks and layers. Motivated by this observation, we reformulate the multimodal instruction tuning as a bi-objective optimization problem and propose Pareto LoRA, a Pareto-optimal gradient integration strategy that balances the text and image objectives by modulating the gradient direction and strength. Experiments on the CoMM benchmark with Emu2 demonstrate that Pareto LoRA consistently improves multimodal generation balance, achieving up to 44.9% gains in perceptual image quality over vanilla LoRA while maintaining comparable text performance.

Problem

• modality imbalance
• unified multimodal models
• gradient dominance
• parameter-efficient fine-tuning
• multimodal instruction tuning

Innovation

• Pareto LoRA
• modality imbalance
• multimodal instruction tuning
• gradient integration
• parameter-efficient fine-tuning