RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

2026-04-17

AI Summary

Current video-based multimodal large language models (MLLMs) demonstrate strong performance on general comprehension tasks but lack systematic evaluation in rule-based, domain-specific adjudication. This work proposes RefereeBench, the first fine-grained benchmark for automated multi-sport refereeing, encompassing 11 sports, 925 videos, and 6,475 rule-driven question-answer pairs that target five core capabilities: foul judgment, classification, reasoning, entity awareness, and temporal localization. Built upon real-world officiating logic and meticulously human-annotated, the benchmark emphasizes alignment between multimodal evidence and rule application. Experiments reveal that even the strongest closed-source models achieve only around 60% accuracy, while the open-source Qwen3-VL scores 47%, with both exhibiting significant weaknesses in rule interpretation and temporal grounding, often misclassifying legitimate actions as fouls.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While Multimodal Large Language Models (MLLMs) excel at generic video understanding, their ability to support specialized, rule-grounded decision-making remains insufficiently explored. In this paper, we introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees. Spanning 11 sports with 925 curated videos and 6,475 QA pairs, RefereeBench evaluates five core officiating abilities: foul existence, foul and penalty classification, foul and penalty reasoning, entity perception, and temporal grounding. The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence. Extensive evaluations of state-of-the-art MLLMs show that even the strongest models, such as Doubao-Seed-1.8 and Gemini-3-Pro, achieve only around 60% accuracy, while the strongest open-source model, Qwen3-VL, reaches only 47%. These results indicate that current models remain far from being reliable sports referees. Further analysis shows that while models can often identify incidents and involved entities, they struggle with rule application and temporal grounding, and frequently over-call fouls on normal clips. Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding, advancing trustworthy AI-assisted officiating and broader multimodal decision-making.

Problem

• Multimodal Large Language Models
• sports refereeing
• rule-grounded decision-making
• video understanding
• temporal grounding

Innovation

• RefereeBench
• Multimodal Large Language Models
• sports officiating
• rule-grounded reasoning
• temporal grounding