ASSERT: A Measurement Pipeline for GenAI Audits

2026-08-13

AI Summary

This study addresses the challenges of reporting rate volatility and attribution ambiguity caused by metric selection in generative AI auditing. To mitigate these issues, we propose ASSERT, a specification-driven auditing pipeline that innovatively binds reporting rates to explicit measurement specifications. This approach automates the construction of scoring criteria and test case execution for auditing. Empirical case studies demonstrate that metric selection significantly impacts result rankings, thereby validating the efficacy of specification binding. By rendering audit discrepancies attributable and interpretable, ASSERT effectively enhances both the explainability and cross-audit comparability of GenAI evaluation results. Ultimately, this work establishes a novel paradigm for standardized auditing in the generative AI domain.

Abstract

Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.

Problem

• GenAI Audits
• Measurement Choices
• Reported Rate
• Attribution

Innovation

• Specification-driven measurement
• GenAI audits
• Measurement pipeline
• Audit reproducibility
• Behavioral rubric