C2AL: Cohort-Contrastive Auxiliary Learning for Large-scale Recommendation Systems

2025-10-02

AI Summary

Large-scale recommendation systems trained under a global objective implicitly assume user homogeneity, overlooking conditional distribution shifts arising from heterogeneous subpopulations in real-world data—leading to central-tendency bias, weakened representation for long-tail groups, and even attention collapse or neuron deactivation. To address this, we propose a distribution-aware queue-based contrastive auxiliary learning framework: it leverages partially conflicting auxiliary labels to expose subpopulation distribution disparities, enables customized attention layer training, and unifies factorization machines, contrastive learning, and multi-task learning. We further design a novel distribution-aware auxiliary loss to regularize shared representation learning. Evaluated on a billion-scale industrial dataset, our method reduces normalized entropy by 0.16% and improves key metrics for minority groups by over 0.30%, significantly mitigating model bias while maintaining strong overall performance and enhancing fairness for long-tail users.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Search and Optimization: Distributed Search
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

Training large-scale recommendation models under a single global objective implicitly assumes homogeneity across user populations. However, real-world data are composites of heterogeneous cohorts with distinct conditional distributions. As models increase in scale and complexity and as more data is used for training, they become dominated by central distribution patterns, neglecting head and tail regions. This imbalance limits the model's learning ability and can result in inactive attention weights or dead neurons. In this paper, we reveal how the attention mechanism can play a key role in factorization machines for shared embedding selection, and propose to address this challenge by analyzing the substructures in the dataset and exposing those with strong distributional contrast through auxiliary learning. Unlike previous research, which heuristically applies weighted labels or multi-task heads to mitigate such biases, we leverage partially conflicting auxiliary labels to regularize the shared representation. This approach customizes the learning process of attention layers to preserve mutual information with minority cohorts while improving global performance. We evaluated C2AL on massive production datasets with billions of data points each for six SOTA models. Experiments show that the factorization machine is able to capture fine-grained user-ad interactions using the proposed method, achieving up to a 0.16% reduction in normalized entropy overall and delivering gains exceeding 0.30% on targeted minority cohorts.

Problem

• Addresses data heterogeneity in large-scale recommendation systems
• Mitigates model bias towards dominant distribution patterns
• Enhances learning for minority user cohorts through auxiliary regularization

Innovation

• Uses cohort-contrastive auxiliary learning for recommendation
• Leverages conflicting auxiliary labels to regularize representations
• Customizes attention layers to preserve minority cohort information