From Prediction to Incrementality: Causal Optimization for Large-Scale Targeting and Recommendation

2026-08-10

AI Summary

This work addresses the limitation of conventional recommendation and delivery systems that rely on predictive scores, which often misallocate resources to users who would naturally convert without intervention, thereby failing to optimize true causal uplift. To overcome this, the authors propose an end-to-end decision-centric framework that directly optimizes individual-level causal effects under global business constraints. The framework uniquely integrates three components within an industrial-scale system: a Transformer-based causal neural network for effect estimation, a Bayesian neural bandit for uncertainty-aware exploration, and a dual-based large-scale linear programming solver for constrained global allocation. It further supports sequential context modeling and multi-objective conditional scoring. Online A/B tests in LinkedIn Feed marketing campaigns demonstrate a statistically significant 7.20% improvement in long-term value metrics, validating the effectiveness and feasibility of large-scale causal optimization under real-world operational constraints.

Abstract

Large-scale targeting and recommendation systems are typically built around predictive scores fed into heuristic or local allocation. When the business goal is incremental impact, as in marketing campaigns, incentives, and notifications, this paradigm systematically misallocates resources toward users who would have acted anyway. We present a decision-centric framework that instead optimizes causal effects under global constraints, aligning three components under a single objective: a causal neural network with a Transformer backbone for individual treatment-effect estimation, a Bayesian neural-bandit layer for uncertainty-aware exploration, and a dual-based large-scale linear-programming layer for constrained allocation. The framework also supports sequential context and multi-outcome, attribute-conditioned scoring through a Transformer encoder and outcome embeddings. We evaluate it with offline simulations on a public bandit dataset, targeted architectural ablations, and an online A/B test on LinkedIn Feed marketing traffic. We also distill production lessons on causal training-data construction and cost and delivery control, which were critical to successful deployment. The end-to-end treatment policy delivered a statistically significant $+7.20\%$ lift in the primary long-term-value metric, demonstrating the feasibility of production-scale causal optimization under business constraints.

Problem

• incrementality
• causal optimization
• large-scale targeting
• recommendation systems
• treatment effect

Innovation

• causal optimization
• treatment effect estimation
• large-scale recommendation
• Bayesian neural bandit
• constrained allocation