Impression Share Prediction: An Offline Evaluation Task for Ranking Systems

2026-08-17

AI Summary

This study addresses the challenge of exposure share shift in offline evaluation for ranking models by introducing the novel task of exposure share prediction. We propose a framework integrating structural causal models with statistical learning, leveraging random forests and encoder-conditional architectures to enable counterfactual prediction of candidate model exposure distributions. This approach effectively bridges the gap between offline metrics and online utility. We theoretically establish the identifiability of counterfactual effects and empirically demonstrate that our method reduces L1 error on training sets by 49% and improves performance during cold-start phases by 22%. These results significantly enhance the reliability of offline evaluation by accurately accounting for distributional shifts in exposure, thereby providing a more robust assessment of ranking model efficacy prior to deployment.

Abstract

Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.

Problem

• Offline Evaluation
• Impression Share Prediction
• Ranking Systems
• Counterfactual Inference
• Downstream Utility

Innovation

• Impression Share Prediction
• Offline Evaluation
• Structural Causal Model
• Counterfactual Inference
• Encoder-conditioned Architecture