JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation

2025-05-19

AI Summary

This paper addresses the lack of a formal task definition, unified evaluation paradigm, and multimodal benchmark dataset for Just-in-Time Information Recommendation (JIR). To this end, it proposes the first systematic JIR framework: (1) It formally defines JIR mathematically and introduces a three-dimensional evaluation metric encompassing demand inference accuracy, recommendation timeliness, and interference suppression capability. (2) It designs a human-in-the-loop modeling approach to characterize user information need distributions and incorporates a multi-round, multi-entity validation mechanism to enhance evaluation objectivity and generalizability. (3) It releases JIR-Arena—the first open-source, multimodal JIR benchmark—supporting reproducible retrieval evaluation based on static knowledge snapshots and real-time streaming input processing. Baseline experiments confirm the effectiveness of demand simulation but reveal room for improvement in recall and retrieval quality.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Just-in-time Information Recommendation (JIR) is a service designed to deliver the most relevant information precisely when users need it, , addressing their knowledge gaps with minimal effort and boosting decision-making and efficiency in daily life. Advances in device-efficient deployment of foundation models and the growing use of intelligent wearable devices have made always-on JIR assistants feasible. However, there has been no systematic effort to formally define JIR tasks or establish evaluation frameworks. To bridge this gap, we present the first mathematical definition of JIR tasks and associated evaluation metrics. Additionally, we introduce JIR-Arena, a multimodal benchmark dataset featuring diverse, information-request-intensive scenarios to evaluate JIR systems across critical dimensions: i) accurately inferring user information needs, ii) delivering timely and relevant recommendations, and iii) avoiding irrelevant content that may distract users. Developing a JIR benchmark dataset poses challenges due to subjectivity in estimating user information needs and uncontrollable system variables affecting reproducibility. To address these, JIR-Arena: i) combines input from multiple humans and large AI models to approximate information need distributions; ii) assesses JIR quality through information retrieval outcomes using static knowledge base snapshots; and iii) employs a multi-turn, multi-entity validation framework to improve objectivity and generality. Furthermore, we implement a baseline JIR system capable of processing real-time information streams aligned with user inputs. Our evaluation of this baseline system on JIR-Arena indicates that while foundation model-based JIR systems simulate user needs with reasonable precision, they face challenges in recall and effective content retrieval. To support future research in this new area, we fully release our code and data.

Problem

• Defining JIR tasks and evaluation metrics formally
• Creating a benchmark dataset for JIR systems
• Addressing challenges in JIR system reproducibility

Innovation

• Mathematical definition of JIR tasks
• Multimodal benchmark dataset JIR-Arena
• Baseline JIR system evaluation