CHANCERY: Evaluating corporate governance reasoning capabilities in language models
Evaluating large language models’ (LLMs) legal reasoning capabilities in corporate governance—particularly for charter compliance assessment—lacks standardized, domain-specific benchmarks. Method: We introduce CHANCERY, the first benchmark dedicated to charter compliance judgment, constructed from 24 core governance principles and 79 real-world, cross-industry corporate charters. It formalizes compliance verification as a binary classification task: determining whether proposals by executives, boards, or shareholders conform to charter provisions. Crucially, we pioneer rule-constrained logical reasoning as the evaluation core, deploying ReAct and CodeAct reasoning agents for fine-grained, traceable assessment. Contribution/Results: State-of-the-art LLMs achieve only 64.5%–75.2% accuracy; reasoning agents improve performance to 76.1%–78.1%, yet reveal persistent bottlenecks in handling cross-referenced clauses and inferring implicit obligations. CHANCERY fills a critical gap in legal reasoning evaluation for corporate governance and provides a foundational benchmark and diagnostic toolkit for trustworthy AI deployment in corporate law.