Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

2026-04-24

AI Summary

This study addresses the challenge of reliably evaluating business ideas generated by large language models, where expert judgments often exhibit structural disagreement due to multidimensional evaluation criteria. To investigate this issue, the authors construct PBIG-DATA, a dataset comprising 300 patent-derived business ideas and 3,000 expert ratings, and systematically compare aggregate versus personalized automated evaluators. Results demonstrate that expert disagreement is structural rather than random, and that personalized evaluators—trained on a target expert’s historical ratings—significantly outperform aggregate approaches, achieving higher fidelity to individual expert judgments across all evaluation dimensions. Notably, model reasoning similarity correlates significantly with human agreement only under personalized settings. These findings reveal that evaluation models trained on unified labels are fragile in diverse assessment contexts and underscore the necessity of evaluator-conditioned designs for robust automated assessment.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment.

Problem

• business idea evaluation
• expert disagreement
• aggregate judge
• personalized judge
• LLM-generated ideas

Innovation

• personalized evaluation
• expert disagreement
• business idea assessment
• LLM judge
• PBIG-DATA