CAIRO: Decoupling Order from Scale in Regression

2026-02-16

AI Summary

Standard regression methods jointly learn ranking and scale, rendering them vulnerable to outliers and heavy-tailed noise. This work proposes a decoupled two-stage framework: first, a scale-invariant ranking loss—such as RankNet or Gini covariance—is employed to learn the optimal ordering of the scoring function; theoretically, this “ranking-optimal” objective recovers the ordering of the true conditional mean. Subsequently, isotonic regression is applied to recover the target scale, and monotonic calibration ensures consistency of the full regression function. Implemented with neural networks, the approach matches state-of-the-art tree ensemble methods on tabular data and significantly outperforms conventional regression objectives under heavy-tailed or heteroscedastic noise.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Standard regression methods typically optimize a single pointwise objective, such as mean squared error, which conflates the learning of ordering with the learning of scale. This coupling renders models vulnerable to outliers and heavy-tailed noise. We propose CAIRO (Calibrate After Initial Rank Ordering), a framework that decouples regression into two distinct stages. In the first stage, we learn a scoring function by minimizing a scale-invariant ranking loss; in the second, we recover the target scale via isotonic regression. We theoretically characterize a class of"Optimal-in-Rank-Order"objectives -- including variants of RankNet and Gini covariance -- and prove that they recover the ordering of the true conditional mean under mild assumptions. We further show that subsequent monotone calibration guarantees recovery of the true regression function. Empirically, CAIRO combines the representation learning of neural networks with the robustness of rank-based statistics. It matches the performance of state-of-the-art tree ensembles on tabular benchmarks and significantly outperforms standard regression objectives in regimes with heavy-tailed or heteroskedastic noise.

Problem

• regression
• outliers
• heavy-tailed noise
• scale-invariance
• ranking

Innovation

• decoupled regression
• rank-based learning
• isotonic regression
• scale-invariant loss
• robust regression