Democratizing News Recommenders: Modeling Multiple Perspectives for News Candidate Generation with VQ-VAE

2025-08-19

AI Summary

Existing news recommendation systems over-rely on click behavior, compromising normative diversity—i.e., fair coverage across ideological or thematic perspectives—and commonly deploy diversity mechanisms only *after* candidate generation, limiting their effectiveness. Method: We propose the first approach to inject *configurable diversity* early in candidate generation: (1) learning discrete, multi-dimensional perspective representations of news articles via a Vector Quantized Variational Autoencoder (VQ-VAE); (2) modeling user preferences over these perspectives using a decoder-only language model; and (3) enabling democratic fairness in retrieval via dynamic inversion of specific dimensions in the query vector. Contribution/Results: Evaluated on the MIND dataset, our method significantly improves candidate set novelty, perspective diversity, and serendipitous exposure. It enables flexible trade-offs between personalization and informational breadth, while systematically enhancing coverage of democracy-relevant perspectives—thereby advancing normative fairness in news recommendation.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Current News Recommender Systems based on past clicks are designed for engagement, but come at the cost of limiting diversity in the suggested content. While diversity-aware algorithms exist, they suffer from two major limitations. First, they fail to account for normative diversity, which requires fair access to a broad range of perspectives. Second, they typically apply diversity late in the system's pipeline, after a lot of content has already been filtered out. Both limitations confine their effectiveness and prevent them from promoting true normative diversity in news recommendations. We propose Aspect-Aware Candidate Generation (A2CG) to address these limitations. Our framework introduces diversity into the earliest pipeline stage and uses a configurable mechanism to align diversity with specific democratic goals. A2CG represents each news article using multiple aspects of perspectives (e.g., sentiment, political leaning, frame) and uses a Vector Quantized Variational Autoencoder (VQ-VAE) to create a discrete, multi-faceted representation. A decoder-only model then learns user preferences over these aspect codes. We then inject diversity directly by reversing the sign on some of the query vector's aspects during the candidate retrieval process, ensuring a more diverse set of candidates. Our method, evaluated on the MIND dataset, enables a flexible trade-off between personalization and diversity early in the recommendation pipeline. It also generates more novel, diverse, and serendipitous candidates while effectively taking into account aspects that strengthen democratic values. These empirical results make it a promising approach for downstream democratized news recommendation systems.

Problem

• Addressing limited diversity in news recommendation systems
• Incorporating normative diversity early in recommendation pipeline
• Balancing personalization with democratic values in news selection

Innovation

• VQ-VAE creates discrete multi-faceted news representations
• Aspect-aware candidate generation injects diversity early
• Configurable mechanism aligns diversity with democratic goals