PIT-SUN: A Deployable Empirical Marginal Transform Framework with Expectation-Consistent Recovery for Regression in Recommender Systems

2026-07-09

AI Summary

This work addresses the challenges in recommender systems posed by long-tailed, zero-inflated, and multimodal target variables, which cause standard MSE loss to suffer from mean collapse and tail shrinkage. Existing nonlinear transformation methods struggle to simultaneously achieve bounded compression and expectation consistency. To overcome these limitations, we propose the PIT-SUN framework, which leverages the Probability Integral Transform (PIT) to construct bounded normal score coordinates and integrates an empirical quantile table with a multiplicative SUN-based unbiased recovery mechanism. This approach enables stable estimation of conditional expectations in the original space without requiring an explicit inverse transform. PIT-SUN uniquely unifies bounded tail compression, expectation-consistent recovery, and lightweight deployability while effectively resolving key practical issues such as coordinate selection, inverse lookup, recovery basis, and drift monitoring under sparse distributions. Experiments demonstrate that PIT-SUN significantly improves point prediction accuracy, calibration, and ranking quality across synthetic, public, and industrial-scale datasets with minimal deployment overhead.

Abstract

Estimating original-space conditional expectations is central to value-driven recommender systems, including dwell time, GMV, and LTV forecasting. Standard MSE is expectation-consistent in principle, but its gradients become unstable on heavy-tailed, zero-inflated, and multimodal targets, causing mean collapse and tail shrinkage. Target transformation alleviates this scale conflict, yet any useful nonlinear marginal transform loses expectation consistency under direct inversion. This is not an implementation oversight: a direct inverse-transform estimator is universally expectation-consistent only when the inverse transform is affine, which cannot simultaneously provide bounded tail compression. Existing conditionally linear recovery methods restore expectation consistency, but still leave open which coordinate, inverse lookup, recovery base, and deployment monitor should be selected for sparse complex marginals. We propose \textbf{P}robability-\textbf{I}ntegral-\textbf{TranS}formed \textbf{Un}biased recovery (\textbf{PIT-SUN}), a deployable empirical marginal recovery framework. PIT-SUN uses one empirical marginal table to define a bounded normal-score coordinate, its inverse-quantile lookup, a variance-controlled recovery base, and drift monitoring, then applies multiplicative SUN recovery to estimate the original-space expectation instead of directly inverting transformed predictions. Experiments on synthetic distributions, public benchmarks, large-scale industrial datasets, and online deployment show robust improvements in point accuracy, calibration, and ranking quality with lightweight deployment overhead.

Problem

• expectation consistency
• marginal transform
• heavy-tailed distribution
• recommendation systems
• conditional expectation estimation

Innovation

• expectation-consistent recovery
• marginal transform
• probability integral transform
• heavy-tailed regression
• recommendation systems