Verifiable User Simulation for Search and Recommendation Systems

2026-06-12

AI Summary

This work addresses the lack of transparency and verifiability in existing large language model (LLM)-driven user simulators, which are prone to biases stemming from user background assumptions. The authors propose the first verifiable user simulator framework, conceptualizing the simulator as an auditable engineering component composed of seven key elements: structured Persona, task-aware Contract, human-aligned Execution, traceable Trace, profile-consistency Verification, structured Feedback, and continuous Refinement. By integrating an end-to-end auditing mechanism that combines LLMs, task contracts, bias detection, and feedback-driven optimization, the framework significantly enhances the credibility, fairness, and interpretability of simulated user behaviors. Empirical evaluations in recommendation list assessment and search query generation demonstrate the simulator’s high fidelity and its effectiveness in mitigating demographic biases.

Abstract

Large-language-model (LLM) based user simulation is increasingly adopted for evaluating search engines, recommender systems, and retrieval-augmented generation pipelines, yet most simulators remain opaque: it is difficult to determine why a simulated user made a particular choice or whether that choice is consistent with the intended user profile. Compounding this, recent research shows that LLMs can produce biased or discriminatory responses depending on user background characteristics such as language, education level, and cultural context, raising concerns about the equitable treatment of minority and disadvantaged groups. This half-day, in-person tutorial introduces a proposed design-and-audit framework that treats a user simulator as a verifiable engineering artefact composed of seven auditable components - structured Persona, task-aware Contract, matched human-vs-agent Execution, auditable Trace, persona-aligned Verification, structured Feedback, and a Refinement loop that updates personas and contracts. Through two hands-on mini-labs on recommendation-list evaluation and search-query formulation, participants will inspect simulator behaviour end-to-end, distinguish diagnostic discrepancy analysis from statistical validation, and apply checks for fidelity, credibility, and demographic bias. The tutorial targets information retrieval and recommender systems researchers and practitioners interested in user behaviour simulation and responsible AI.

Problem

• user simulation
• verifiability
• bias
• fairness
• recommender systems

Innovation

• verifiable user simulation
• auditable components
• demographic bias mitigation
• responsible AI
• retrieval evaluation