What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

2026-08-17

AI Summary

This study addresses the prevalent "rule blindness" deficiency in existing compliance detectors, which rely on superficial features rather than actual regulatory rules. We propose a training-free Internal Compliance Score (ICS) and a Cross-Rule Counterfactual Audit Protocol to investigate this limitation. Through multidimensional verification involving activation probing and guard models, we confirm the ubiquity of rule blindness across systems. Our findings indicate that while ICS does not surpass performance baselines, it enables cost-effective auditing of internal states across multiple models. Furthermore, stepwise reasoning mechanisms are validated as a critical pathway for overcoming this defect. Ultimately, this work reveals the intrinsic limitations of current compliance detection approaches and establishes a novel paradigm for constructing interpretable and robust compliance auditing frameworks.

Abstract

Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector's verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Auditing at scale requires a retraining-free detector, so we introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold ICS to the same scrutiny as the guards it audits: a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.

Problem

• Rule Blindness
• Compliance Detectors
• Activation Probes
• Guard Models
• Regulatory Compliance

Innovation

• Rule Blindness
• Internal Compliance Score
• Training-free Audit
• Crossed-rule Benchmark
• Activation Probes