Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)

2025-05-16

AI Summary

Evaluating interactive information access (IA) systems is hindered by limited access to real users and ethical constraints in information retrieval research. Method: This work systematically develops a simulation methodology and toolchain for interactive IA, integrating multi-agent user modeling, behavioral simulation, synthetic interaction log generation, and a reproducible evaluation metrics framework. Contribution/Results: It introduces the first simulation-driven paradigm for IA evaluation, substantially reducing experimental complexity, enhancing result reproducibility, and deepening understanding of user cognition and interaction dynamics. The project has catalyzed the前瞻性 inclusion of simulation tasks in major international evaluation forums—including TREC and CLEF—fostered cross-institutional collaboration, facilitated initial community consensus on simulation methodology standards and validation criteria, and launched the world’s first joint IA simulation evaluation initiative.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Humans and AI: Interaction Techniques and Devices

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central IA research and evaluation questions, especially around interactive settings when real users are unavailable, or their participation is impossible due to ethical reasons. In addition, simulations in IA can help contribute to a better understanding of users, reduce complexity of evaluation experiments, and improve reproducibility. Building on recent developments in methods and toolkits, the second iteration of our Sim4IA workshop aims to again bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.

Problem

• Exploring simulations for information access research
• Addressing ethical and practical user participation limitations
• Improving evaluation reproducibility and user understanding

Innovation

• Simulations for interactive IA research
• Toolkits to reduce evaluation complexity
• Workshop to discuss future IA perspectives