Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software

2026-04-12

AI Summary

This study addresses a critical gap between compliance and effectiveness in current auditing standards—such as ASB 018—whose reliance on ambiguous language and undefined terminology obscures the potential risks associated with the use of probabilistic genotyping software in criminal justice. Through a qualitative content analysis comparing the standard’s text with five real-world audit reports, this work demonstrates for the first time that audits deemed compliant often fail to delineate the boundaries of software application. The research attributes this disconnect to structural deficiencies in the standard itself and offers concrete recommendations for revising auditing frameworks and evaluating their practical efficacy. These contributions provide both theoretical insight and actionable guidance for enhancing the governance of forensic technologies within the justice system.

Technology Category

• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Responsible Web: Audits of web technologies, standards, platforms, and applications
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Cyber-crime defenses and forensics

Abstract

AI governance efforts increasingly rely on audit standards: agreed-upon practices for conducting audits. However, poorly designed standards can hide and lend credibility to inadequate systems. We explore how an audit standard's design influences its effectiveness through a case study of ASB 018, a standard for auditing probabilistic genotyping software -- software that the U.S. criminal legal system increasingly uses to analyze DNA samples. Through qualitative analysis of ASB 018 and five audit reports, we identify numerous gaps between the standard's desired outcomes and the auditing practices it enables. For instance, ASB 018 envisions that compliant audits establish restrictions on software use based on observed failures. However, audits can comply without establishing such boundaries. We connect these gaps to the design of the standard's requirements such as vague language and undefined terms. We conclude with recommendations for designing audit standards and evaluating their effectiveness.

Problem

• audit standards
• probabilistic genotyping
• compliance gap
• AI governance
• forensic software

Innovation

• audit standards
• probabilistic genotyping
• AI governance
• compliance gap
• qualitative analysis