LatentAudit: Real-Time White-Box Faithfulness Monitoring for Retrieval-Augmented Generation with Verifiable Deployment

2026-04-06

AI Summary

This work addresses the challenge of real-time faithfulness evaluation in retrieval-augmented generation (RAG) systems by proposing a white-box monitoring approach that requires no additional discriminative model. The method introduces a novel quadratic rule based on the Mahalanobis distance between residual stream activations in the latter layers of the generative model and the retrieved evidence representations, leveraging the geometric structure of residual streams as a signal for faithfulness. It further integrates 16-bit fixed-point quantization with Groth16 zero-knowledge proofs to enable cross-architecture stability and publicly verifiable deployment without exposing model weights or internal activations. Experiments demonstrate an AUROC of 0.942 on PubMedQA with only 0.77ms latency overhead; under diverse models and stress tests, AUROC ranges from 0.9142 to 0.9815, and quantization preserves 99.8% of original performance.

Abstract

Retrieval-augmented generation (RAG) mitigates hallucination but does not eliminate it: a deployed system must still decide, at inference time, whether its answer is actually supported by the retrieved evidence. We introduce LatentAudit, a white-box auditor that pools mid-to-late residual-stream activations from an open-weight generator and measures their Mahalanobis distance to the evidence representation. The resulting quadratic rule requires no auxiliary judge model, runs at generation time, and is simple enough to calibrate on a small held-out set. We show that residual-stream geometry carries a usable faithfulness signal, that this signal survives architecture changes and realistic retrieval failures, and that the same rule remains amenable to public verification. On PubMedQA with Llama-3-8B, LatentAudit reaches 0.942 AUROC with 0.77,ms overhead. Across three QA benchmarks and five model families (Llama-2/3, Qwen-2.5/3, Mistral), the monitor remains stable; under a four-way stress test with contradictions, retrieval misses, and partial-support noise, it reaches 0.9566--0.9815 AUROC on PubMedQA and 0.9142--0.9315 on HotpotQA. At 16-bit fixed-point precision, the audit rule preserves 99.8% of the FP16 AUROC, enabling Groth16-based public verification without revealing model weights or activations. Together, these results position residual-stream geometry as a practical basis for real-time RAG faithfulness monitoring and optional verifiable deployment.

Problem

• Retrieval-Augmented Generation
• Hallucination
• Faithfulness Monitoring
• Evidence Support
• Real-Time Audit

Innovation

• LatentAudit
• residual-stream geometry
• Mahalanobis distance
• retrieval-augmented generation
• verifiable deployment