MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents

2026-05-07

AI Summary

Current evaluations of tool-use compliance in large language model (LLM) agents rely on human-curated benchmarks or LLM-based judges, which suffer from limited scalability and insufficient reliability under complex, long-horizon protocols. This work proposes MANTRA, a framework that, for the first time, automatically generates machine-verifiable formal compliance benchmarks from arbitrary-domain, long-form natural language manuals and tool specifications. MANTRA integrates symbolic world modeling, SMT-solver-driven consistency verification, trajectory-level compliance checking, and a structured repair loop, enabling tunable task complexity and fine-grained failure diagnosis. Evaluated on a benchmark suite spanning six domains and 285 tasks derived from over 50 pages of documentation, the approach substantially enhances the scalability, reliability, and granularity of compliance assessment.

Abstract

Tool-using large language model (LLM) agents are increasingly deployed in settings where their reliable behavior is governed by strict procedural manuals. Ensuring that such agents comply with the rules from these manuals is challenging, as they are typically written for humans in natural language while agent behavior manifests as an execution trace of tool calls. Existing evaluations of LLM agents rely on manually constructed benchmarks or LLM-based judges, which either do not scale or lack reliability for complex, long-horizon manuals. To overcome these limitations, we present MANTRA, a framework for automatically synthesizing machine-checkable compliance benchmarks from natural-language manuals and tool schemas. MANTRA independently generates (i) a symbolic world model capturing procedural dependencies, and (ii) a set of trace-level compliance checks for a given task, and validates their consistency using SMT solving. A structured repair loop resolves inconsistencies, requiring human intervention only as a fallback. %This yields benchmarks that are formally validated. Importantly, MANTRA supports arbitrary domains and long procedural manuals, and provides a tunable notion of task complexity which is utilized to automatically derive challenging tasks accompanying compliance checks. Using MANTRA, we build a new benchmark suite with 285 tasks across 6 domains scaling to 50+ page manuals with minimal human effort. Empirically, we show that the compliance checks are richer with stronger constraint enforcement compared to existing benchmarks. Additionally, the granularity of the checks can be used for debugging the agents' failure modes. These results demonstrate that combining automated benchmark generation with formally grounded validation methods enables scalable and reliable benchmarking of tool-using agents.

Problem

• compliance benchmarking
• tool-using LLM agents
• procedural manuals
• execution trace validation
• scalable evaluation

Innovation

• SMT solving
• compliance benchmarking
• tool-using LLM agents
• symbolic world model
• automated benchmark synthesis