IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

2026-02-26

AI Summary

Commercial large language models are often deployed as black-box APIs, exposing users to economically motivated attacks such as model substitution, excessive quantization, or inflated token usage reporting, while offering no means to verify the correctness of inference or billing integrity. This work proposes the first efficient auditing framework that requires neither trusted hardware nor internal model access, leveraging verifiable computation and selective auditing strategies to cryptographically validate both inference execution and token consumption. The approach supports both dense and mixture-of-experts architectures, achieving strong detection guarantees with less than 1% throughput overhead. Experimental results demonstrate its ability to reliably distinguish between legitimate and malicious executions, thereby effectively safeguarding user interests.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.

Problem

• LLM auditing
• black-box API
• verifiable computation
• token overbilling
• model substitution

Innovation

• verifiable computation
• LLM auditing
• model substitution detection
• token overbilling
• black-box API security