WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

2026-09-10

AI Summary

本文提出WinSyn，一种自动生成反映真实工作场景的合成数据集的管道，用于评估企业问答系统，强调信息分散和自然查询，揭示现有模型改进空间。

Abstract

Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflict- ing emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings. In this work, we introduce an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. Our method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. The data emphasizes ambiguity, distributed information, and naturally occurring queries. To validate the pipeline, we evaluate few standard agentic baselines on our datasets using the latest frontier models. We find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement. These findings suggest that more work remains to be done for enterprise deployment and underscore the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.

Problem

• Enterprise Question-Answering
• Complexity of Enterprise Data
• Realistic Evaluation

Innovation

• Automated Pipeline
• Realistic Enterprise Data
• Retrieval-Augmented Generation
• Deep Research