Model Agreement via Anchoring

2026-02-26

AI Summary

This work addresses the problem of controlling the expected squared disagreement between predictions from two independently trained machine learning models. To this end, the authors propose a general “anchoring” analysis technique that aligns model outputs to their average, enabling a unified derivation of upper bounds on prediction disagreement across diverse algorithms. For the first time, this framework simultaneously encompasses stacked aggregation, gradient boosting, neural architecture search, and fixed-depth regression trees. Under strongly convex losses—including multidimensional regression—the analysis establishes that the disagreement between models is theoretically guaranteed to converge to zero as natural training parameters (such as the number of models, iteration count, architecture size, or tree depth) increase.

Technology Category

• Machine Learning: Ensemble Methods
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Numerous lines of aim to control $\textit{model disagreement}$ -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on $\textit{anchoring}$ to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models $k$ being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations $k$) (3) neural network training with architecture search (where disagreement is driven to 0 with the size $n$ of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth $d$ of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss -- but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.

Problem

• model disagreement
• real-valued prediction
• independent training
• prediction divergence
• machine learning models

Innovation

• model disagreement
• anchoring
• stacked aggregation
• gradient boosting
• strongly convex loss