xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

📅 2025-06-16
📈 Citations: 0
✨ Influential: 0
📄 PDF
🤖 AI Summary
Existing AI evaluation benchmarks focus on isolated technical competencies, failing to capture real-world economic value in professional settings. To address this, we propose xbench—a profession-aligned, dynamic evaluation paradigm grounded in authentic business contexts such as recruitment and influencer marketing. Tasks are co-designed with domain experts and integrate domain-specific modeling, real-world operational data (e.g., 50 headhunting assignments, 836 influencer profiles), and multidimensional productivity metrics. We introduce the first economic-value-centric, occupation-level assessment framework, featuring the Technology-Market Fit (TMF) metric—a predictive indicator for quantifying alignment between AI capabilities and market needs—enabling longitudinal capability tracking and adaptive benchmark evolution. We release two open-source benchmarks and empirically validate strong correlations between TMF scores and actual business outcomes across mainstream AI agents (xbench.org).

Technology Category

Application Category

📝 Abstract
We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.
Problem

Research questions and friction points this paper is trying to address.

Bridging AI agent capabilities and real-world productivity gaps
Evaluating economic value of AI in professional settings
Tracking agent productivity scaling with industry-aligned metrics
Innovation

Methods, ideas, or system contributions that make the work stand out.

Dynamic profession-aligned evaluation suite
Metrics correlating with productivity value
Real-world tasks from industry professionals
🔎 Similar Papers
No similar papers found.
Authors to Follow
Kaiyuan Chen
Kaiyuan Chen
Bytedance
LLMScaling LawAI4WeatherVideo Generation
Y
Yixin Ren
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University
Y
Yang Liu
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University
X
Xiaobo Hu
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University
H
Haotong Tian
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University
Tianbao Xie
Tianbao Xie
University of Hong Kong
Artificial IntelligenceDeep LearningNatural Language Processing
Fangfu Liu
Fangfu Liu
Tsinghua University
Computer Vision3D VisionMachine Learning
H
Haoye Zhang
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University
Hongzhang Liu
Hongzhang Liu
Rutgers
Cloud ComputingVehicular NetworksSensor Networks
Y
Yuan Gong
Carnegie Mellon University, Fudan University, Imperial College London, Massachusetts Institute of Technology, National University of Singapore, Peking University, Shanghai Jiao Tong University, Stanford University, The Chinese University of Hong Kong (Shenzhen), The Ohio State University, Tsinghua University, University of Chinese Academy of Sciences, University of Oxford, University of Pennsylvania, University of Science and Technology of China, University of Sydney, University of Toronto, Yale University