Benchmarking Real-Time Question Answering via Executable Code Workflows

2026-03-16

AI Summary

Existing question-answering benchmarks are predominantly static, limiting their ability to evaluate models’ capacity to adapt to the dynamic evolution of real-world knowledge. This work proposes RT-QA, the first dynamic QA evaluation framework that integrates executable code workflows with a self-repair mechanism: autonomous agents generate and execute web-scraping and DOM-parsing code at evaluation time to retrieve answers in real time, automatically adapting to changes in webpage structures. The framework encompasses 320 Chinese questions spanning 12 domains and reveals two prevalent failure modes among current large language models in real-time retrieval—“lazy retrieval” and “temporal confusion.” Experimental results show that even the strongest model achieves only 46% accuracy, highlighting significant limitations in dynamic, real-world settings.

Abstract

Retrieving real-time information is a fundamental capability for search-integrated agents in real-world applications. However, existing benchmarks are predominantly static and therefore fail to capture the temporal dynamics of information and the continuously evolving nature of real-world knowledge. To address this limitation, we propose RT-QA, a dynamic evaluation framework that leverages executable code workflows to retrieve up-to-date answers at evaluation time. Specifically, we construct an agent-driven pipeline that autonomously generates code for web crawling and DOM-based answer extraction to produce real-time ground truth. To ensure robust evaluation over time, the pipeline further incorporates a self-repair mechanism to adapt to changes in web page structures. RT-QA spans 12 domains (e.g., Finance, Sports) with 320 Chinese questions categorized into three difficulty levels. Extensive evaluations of state-of-the-art models (e.g., GPT-5.2, GLM-4.7) reveal significant limitations in real-time adaptability: even the best models achieve only 46% accuracy. Our analysis highlights two primary failure modes: (1) Lazy Retrieval, where agents rely on search snippets instead of deeply scanning specific websites for information (20% of failures); and (2) Temporal Confusion, a cognitive error where agents retrieve a historical date (e.g., an event in 2024) and fail to re-anchor to the current time (2026) for subsequent reasoning. These findings suggest that future agents require not just better retrieval strategies, but robust temporal state management.

Problem

• real-time question answering
• temporal dynamics
• dynamic benchmarking
• information retrieval
• temporal reasoning

Innovation

• real-time question answering
• executable code workflows
• dynamic benchmarking
• self-repair mechanism
• temporal state management