IIRSim Studio: A Dashboard for User Simulation

2026-04-25

AI Summary

This work addresses the limitations of existing information retrieval user simulation frameworks, which are predominantly code-centric libraries lacking integrated infrastructure for experiment design, execution, and sharing, thereby imposing high usability barriers and poor reproducibility. To overcome these challenges, the authors propose a web-based visual workbench that enables users to orchestrate simulation workflows through a graphical interface. The system innovatively integrates Git-driven component versioning, an experiment-package provenance model, and a shared-task mechanism. Built with a web frontend, containerized deployment, and runtime environment injection, the platform has successfully reproduced the Sim4IA microtask and offers both hosted services and portable deployment options, significantly lowering entry barriers and enhancing experimental reproducibility.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio, a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment.

Problem

• user simulation
• information retrieval
• reproducibility
• experiment workflow
• simulation framework

Innovation

• user simulation
• information retrieval
• reproducibility
• visual workflow
• experiment provenance