Probabilistic Model Checking of Autoregressive Neural Sequence Models

2026-09-01

AI Summary

该研究使用概率模型检测方法解决自回归神经序列模型在部署时的约束违反概率和输入满足领域需求比例的问题。

Abstract

Test-set accuracy is silent on two issues that matter when deploying autoregressive neural sequence models: how much probability mass the system under test (SUT) places on constraint-violating alternatives that are reachable under sampling and what fraction of the input population satisfies a domain requirement. We answer both with probabilistic model checking. The pipeline extracts a discrete-time Markov chain (DTMC) from the SUT's token-by-token generation, verifies formal PCTL specifications with the PRISM model checker, and aggregates the per-input verdicts into a coverage curve over the input space. A soundness theorem establishes the DTMC as an under-approximation, so every verdict yields a certified interval on the SUT's true reachability probability. The coverage built from those verdicts is, therefore, conservative by construction. A counterexample-guided abstraction refinement (CEGAR) loop adaptively tightens the interval, and a maximum-likelihood algorithm extracts the most probable falsifying trace. Two case studies exercise the pipeline. On a GPT-2 computer-aided process-planning (CAPP) model with 100% test accuracy, the pipeline quantifies the probability mass greedy decoding hides, but that is reachable with sampling; and identifies the smallest training fraction at which an ordering requirement holds population-wide, neither of which test accuracy can report. We then verify the SMILES molecular generator with a 50x larger vocabulary. The only change is an external chemical-validity oracle, and the pipeline identifies the gap between structural completeness and chemical validity.

Problem

• autoregressive neural sequence models
• probability mass
• constraint-violating alternatives
• domain requirement
• sampling

Innovation

• Probabilistic Model Checking
• Discrete-Time Markov Chain (DTMC)
• PRISM Model Checker
• Counterexample-Guided Abstraction Refinement (CEGAR)
• Maximum-Likelihood Algorithm