Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

2026-05-05

AI Summary

This work addresses the challenge of aligning heterogeneous vision-language models (VLMs) in federated learning, where differences in model architectures, local data distributions, and computational resources hinder conventional parameter aggregation, and privacy constraints prohibit sharing raw data. To overcome these limitations, the authors propose the MoR framework, which introduces preference learning into federated alignment for the first time. Each client trains a local reward model based on its own preferences and adaptively fuses them via a learnable routing mechanism within a mixture-of-rewards paradigm. The server then optimizes a base VLM using the GRPO algorithm augmented with KL divergence regularization, without exchanging model architectures, parameters, or data. Experiments demonstrate that MoR significantly outperforms existing federated alignment methods across multiple vision-language benchmarks, exhibiting superior generalization and cross-client adaptability.

Abstract

Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. Federated Learning mitigates this issue by enabling decentralized training, but practical deployments face challenges due to client heterogeneity in computational resources, application requirements, and model architectures. Under extreme model and data heterogeneity, replacing parameter aggregation with preference-based collaboration offers a more suitable interface, as it eliminates the need for direct parameter or data exchange. Motivated by this, we propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneous VLMs. In MoR, each client locally trains a reward model from local preference annotations, capturing specific evaluation signals without exposing raw data. To combine these heterogeneous supervision signals, MoR introduces a Mixture-of-Rewards mechanism with learned routing, which adaptively fuses client reward models according to the input and alignment objective. The server then optimizes a base VLM using GRPO with a KL penalty to a reference model, enabling preference alignment without requiring client models to share architectures or parameters. Experiments on diverse public vision-language benchmarks demonstrate that MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability. Our approach provides a scalable solution for privacy-preserving alignment of heterogeneous VLMs under federated settings.

Problem

• Federated Learning
• Vision-Language Models
• Model Heterogeneity
• Preference Alignment
• Privacy Preservation

Innovation

• Federated Learning
• Vision-Language Models
• Preference Alignment
• Mixture-of-Rewards
• GRPO