Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agents

2026-07-14

AI Summary

This work addresses the limited responsiveness of existing recommender systems to explicit user guidance and the absence of controllability in current evaluation frameworks. To bridge this gap, we propose CtrlBench-Rec, the first standardized benchmark for assessing controllability in recommendation systems, encompassing three core tasks: targeted content discovery, interest profile shaping, and popularity bias mitigation. Our framework employs a multi-agent simulation to model user–system interactions and conducts end-to-end quantitative controllability analysis of mainstream recommendation models using real-world datasets. Experimental results demonstrate that CtrlBench-Rec effectively uncovers system limitations in guiding users toward long-tail content. We further release an open-source toolkit to support future research on controllable recommendation and algorithmic auditing.

Abstract

Recommender systems operate as Black-Boxes, leaving users and regulators unable to steer their outputs toward specific intentions or audit their behavior. This lack of controllability, defined as the system's ability to respond to explicit guidance, remains an unaddressed dimension in existing evaluation paradigms. To fill this gap, we propose CtrlBench-Rec, a collaborative multi-agent framework for systematic assessment of controllability. We formalize three fundamental tasks: target content discovery, interest profile shaping, and popularity bias mitigation, which together measure steerability from explicit commands to implicit representation steering and finally to overcoming algorithmic biases.Extensive experiments on real-world datasets and multiple recommendation models demonstrate that our framework effectively quantifies controllability and exposes critical system bottlenecks, most notably persistent resistance to guiding long tail content. CtrlBench-Rec provides the first standardized toolkit for controllable recommendation research, algorithmic auditing, and user empowerment. Our code is released on https://github.com/caskcsg/CtrlBenchRec.

Problem

• controllability
• recommender systems
• black-box
• algorithmic auditing
• steerability

Innovation

• controllability
• recommender systems
• multi-agent framework
• algorithmic auditing
• steerability