UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

2026-08-16

AI Summary

This study addresses the multifaceted heterogeneity in tasks, modalities, and architectures inherent to federated instruction tuning of vision-language models by proposing a unified privacy-preserving fine-tuning framework. The core innovations include FedCSA, a dynamic weighted subspace aggregation mechanism, and TCoD, a two-stage collaborative distillation strategy. By integrating parameter-efficient adapters with Mixture-of-Experts mutual distillation, this approach effectively overcomes multidimensional heterogeneity constraints. Experimental results demonstrate that the proposed method significantly outperforms existing federated learning baselines across multiple benchmarks, achieving efficient collaboration and robust privacy preservation in complex heterogeneous scenarios.

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, which raises privacy concerns in certain domains (e.g. healthcare). Federated Learning (FL) provides a natural solution by enabling model training without sharing raw data. However, applying FL to VLM instruction tuning is highly challenging. VLMs have substantial parameter scales, and in real-world scenarios, clients exhibit significant heterogeneity in tasks, modalities, and model architectures. Existing methods mainly focus on simplified settings and are unable to handle such multi-dimensional heterogeneous scenarios. In this work, we study federated instruction tuning under joint heterogeneity in tasks, modalities, and model architectures. We propose UniFed-VLM, a unified federated instruction tuning framework for VLMs that addresses multiple types of heterogeneity. It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters with dynamic weighting and compensation to mitigate heterogeneity-induced conflicts; 2) Two-stage Collaborative Distillation (TCoD), which enables effective knowledge transfer across heterogeneous models via a Mutual Distillation Adapter (MDA) and a mixture-of-experts-based distillation strategy. We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance across diverse tasks compared with existing FL methods. The source code is available at: https://github.com/wangpengyu2004/UniFed-VLM.

Problem

• Federated Learning
• Vision-Language Models
• Instruction Tuning
• Heterogeneity

Innovation

• Federated Instruction Tuning
• Vision-Language Models
• Multi-dimensional Heterogeneity
• Subspace Aggregation
• Collaborative Distillation