MCGrad:: Multicalibration at Web Scale

2025-09-24

AI Summary

Existing multi-calibration methods face three critical bottlenecks in industrial settings: reliance on manually defined subgroups, poor scalability, and degradation of key performance metrics—particularly log loss and PRAUC. To address these challenges, we propose MCGrad, a scalable multi-calibration algorithm that requires no explicit specification of protected groups. MCGrad employs a gradient-driven online optimization framework integrated with group-invariance regularization to enable adaptive subgroup calibration. Crucially, it simultaneously ensures calibration across all subgroups while improving global model performance—reducing log loss and increasing PRAUC—thereby harmonizing fairness and accuracy. The method has been deployed at scale across hundreds of production models at Meta. Extensive evaluation on public benchmarks and real-world business applications demonstrates its effectiveness, robustness, and engineering scalability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Multi-modal Vision
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

We propose MCGrad, a novel and scalable multicalibration algorithm. Multicalibration - calibration in sub-groups of the data - is an important property for the performance of machine learning-based systems. Existing multicalibration methods have thus far received limited traction in industry. We argue that this is because existing methods (1) require such subgroups to be manually specified, which ML practitioners often struggle with, (2) are not scalable, or (3) may harm other notions of model performance such as log loss and Area Under the Precision-Recall Curve (PRAUC). MCGrad does not require explicit specification of protected groups, is scalable, and often improves other ML evaluation metrics instead of harming them. MCGrad has been in production at Meta, and is now part of hundreds of production models. We present results from these deployments as well as results on public datasets.

Problem

• Existing multicalibration methods require manual subgroup specification
• Current approaches lack scalability for web-scale applications
• Traditional methods may harm other important model performance metrics

Innovation

• Automatically discovers subgroups without manual specification
• Scalable multicalibration algorithm for web applications
• Improves model performance metrics while maintaining calibration