Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

2026-08-17

AI Summary

This study addresses the challenge of quality auditing for third-party large model inference APIs that lack probability outputs. We propose a probability-agnostic composite black-box auditing framework that reconstructs categorical distributions via stochastic process modeling and zero-bias corrected KL statistics. Furthermore, we introduce joint metrics comprising Average Fidelity Loss (AFL) and Extreme Fidelity Loss (EFL). Experimental results demonstrate that AFL aligns closely with log-probability benchmarks, while EFL effectively reveals performance degradation in long-horizon tasks. By enabling precise fidelity quantification through repeated queries and long-sequence testing alone, this work establishes a reliable paradigm for evaluating black-box AI services without requiring access to internal model probabilities.

Abstract

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.

Problem

• LLM API auditing
• vendor-hosted models
• inference quality verification
• black-box testing

Innovation

• Black-box Audit
• Average Fidelity Loss
• Extreme Fidelity Loss
• Stochastic Routing
• Vendor-Hosted LLM