PetroBench: A Benchmark for Large Language Models in Petroleum Engineering
This study addresses the absence of domain-specific evaluation benchmarks for large language models (LLMs) in petroleum engineering by introducing the first standardized assessment framework encompassing production, reservoir, and drilling engineering, comprising 1,200 multi-format questions. Data quality is ensured through a rigorous three-stage pipeline involving expert review, preprocessing, and quality filtering, followed by validation across multiple models. Systematic evaluations of leading Chinese and English LLMs are conducted under a unified API environment. Results reveal that models perform better on subjective than objective questions, achieving peak accuracies of 65.3% and 74.3% on multiple-choice and true/false items, respectively. Models such as Gemini-1.5-Pro attain overall scores of 72%–74%, with Chinese models excelling in multiple-choice tasks and international models showing slight advantages in short-answer responses. The benchmark demonstrates strong domain relevance, high discriminative power, and reproducibility.