How (Mis)calibrated is Your Federated CLIP and What To Do About It?

2025-12-03

AI Summary

This work investigates calibration degradation in federated learning (FL) settings during fine-tuning of CLIP models, revealing that text prompt tuning severely impairs the reliability of prediction confidence. To address this, we propose FL²oRA—a lightweight, calibration-aware fine-tuning framework tailored for FL. FL²oRA integrates Low-Rank Adaptation (LoRA), a dynamic module selection mechanism, multi-strategy global aggregation, and implicit in-training calibration—without requiring post-hoc correction or explicit calibration modules. It jointly optimizes model accuracy and calibration reliability throughout distributed training. Extensive experiments across multiple vision-language benchmarks demonstrate that FL²oRA consistently improves calibration metrics (e.g., reducing Expected Calibration Error by 32.7% on average), outperforming existing FL calibration methods and effectively mitigating calibration degradation in federated scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior works have examined CLIP calibration in offline settings, the impact of fine-tuning CLIP in a federated learning (FL) setup remains unexplored. In this work, we investigate how FL affects CLIP calibration and propose strategies to improve reliability in this distributed setting. We first analyze Textual Prompt Tuning approaches and show that they degrade calibration metrics when operating under FL. We also evaluate existing in-training calibration techniques across four global aggregation methods, finding that they provide limited improvements. Our results suggest that the key challenge lies not only in how we aggregate or calibrate, but in which components we choose to fine-tune. Motivated by this insight, we propose $ ext{FL}^2 ext{oRA}$, a straightforward LoRA-based approach that naturally improves calibration in FL, and we analyze the factors behind its effectiveness. Experiments on multiple benchmarks demonstrate that $ ext{FL}^2 ext{oRA}$ consistently produces well-calibrated models, reducing the need for explicit calibration procedures. Codes are available at https://github.com/mainaksingha01/FL2oRA.

Problem

• Investigates CLIP calibration in federated learning
• Analyzes degradation of calibration in FL settings
• Proposes FL2oRA to improve calibration naturally

Innovation

• FL2oRA uses LoRA-based fine-tuning for calibration
• It improves federated CLIP reliability without explicit calibration
• Focuses on component selection over aggregation methods