Beyond the Click: A Framework for Inferring Cognitive Traces in Search

2026-02-27

AI Summary

This work addresses a critical limitation in existing user simulators, which reproduce observable behaviors but fail to capture users’ underlying cognitive states—such as confusion or satisfaction—during search interactions. To bridge this gap, the study introduces, for the first time, a cognitively grounded approach to modeling user behavior from interaction logs. Leveraging information foraging theory and human expert judgments, the authors develop a multi-agent system capable of scalably inferring users’ cognitive trajectories from large-scale behavioral data. The proposed method significantly improves performance on downstream tasks, including conversational outcome prediction and user difficulty recovery. Furthermore, the authors release cognitive annotations and accompanying tools for widely used datasets such as AOL and Stack Overflow, establishing a new paradigm for more human-like user simulation and evaluation of retrieval systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

User simulators are essential for evaluating search systems, but they primarily copy user actions without understanding the underlying thought process. This gap exists since large-scale interaction logs record what users do, but not what they might be thinking or feeling, such as confusion or satisfaction. To solve this problem, we present a framework to infer cognitive traces from behavior logs. Our method uses a multi-agent system grounded in Information Foraging Theory (IFT) and human expert judgment. These traces improve model performance on tasks like forecasting session outcomes and user struggle recovery. We release a collection of annotations for several public datasets, including AOL and Stack Overflow, and an open-source tool that allows researchers to apply our method to their own data. This work provides the tools and data needed to build more human-like user simulators and to assess retrieval systems on user-oriented dimensions of performance.

Problem

• user simulators
• cognitive traces
• search systems
• interaction logs
• user behavior

Innovation

• cognitive traces
• user simulation
• Information Foraging Theory
• multi-agent system
• behavior log inference