RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

2026-07-10

AI Summary

This work addresses the challenge of achieving task-aware optimal selection and fusion of heterogeneous recommendation agents—such as collaborative filtering, sequential models, content-based retrievers, and LLM-based rerankers—under cost constraints. The authors propose RouteRec, a framework evaluated under the first offline five-fold protocol that eliminates information leakage. Within this rigorous setting, they systematically compare request-level hard selection against item-level learned aggregation strategies. Empirical results demonstrate that request-level selection is overly coarse, whereas item-level gated aggregation yields substantially superior performance: even lightweight aggregation matches BM25’s HR@10 of 0.254, and full-agent gated aggregation achieves 0.295, significantly outperforming hard selection. These findings highlight the critical potential of fine-grained, item-level fusion in multi-agent recommender systems.

Abstract

Recommender systems increasingly face a choice among heterogeneous agents -- collaborative filters, sequential models, content-based retrievers, and LLM-based rerankers -- yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent. On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0.223 vs. 0.254), and selective LLM escalation does not improve it. The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10 = 0.295 with 70.2\% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space.

Problem

• recommender systems
• agent selection
• cost constraints
• LLM reranker
• heterogeneous agents

Innovation

• agent routing
• item-level aggregation
• recommender systems
• LLM reranker
• cost-constrained evaluation