A doubly composite Chernoff-Stein lemma and its applications

2025-10-07

AI Summary

This paper addresses binary hypothesis testing under doubly composite and strongly correlated hypotheses, focusing on the Stein exponent—the optimal exponential decay rate of the type-II error probability as the type-I error tends to zero. To overcome the failure of the classical Chernoff–Stein lemma under non-i.i.d., asymmetric, and strongly correlated composite settings, we establish the first Chernoff–Stein lemma tailored to two-sided composite strongly correlated hypotheses. We introduce symbol-wise fuzzification, leveraging convexity and weak permutation symmetry to transcend classical symmetry constraints. We derive single-letter characterizations of the Stein exponent for broad classes of composite hypotheses. Furthermore, we develop a “restricted de Finetti reduction” theorem, enabling unified analysis under general convex constraints. Collectively, these results provide a universal theoretical foundation for both classical and quantum hypothesis testing.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction
• Machine Learning: Information Theory

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Given a sequence of random variables $X^n=X_1,ldots, X_n$, discriminating between two hypotheses on the underlying probability distribution is a key task in statistics and information theory. Of interest here is the Stein exponent, i.e. the largest rate of decay (in $n$) of the type II error probability for a vanishingly small type I error probability. When the hypotheses are simple and i.i.d., the Chernoff-Stein lemma states that this is given by the relative entropy between the single-copy probability distributions. Generalisations of this result exist in the case of composite hypotheses, but mostly to settings where the probability distribution of $X^n$ is not genuinely correlated, but rather, e.g., a convex combination of product distributions with components taken from a base set. Here, we establish a general Chernoff-Stein lemma that applies to the setting where both hypotheses are composite and genuinely correlated, satisfying only generic assumptions such as convexity (on both hypotheses) and some weak form of permutational symmetry (on either hypothesis). Our result, which strictly subsumes most prior work, is proved using a refinement of the blurring technique developed in the context of the generalised quantum Stein's lemma [Lami, IEEE Trans. Inf. Theory 2025]. In this refined form, blurring is applied symbol by symbol, which makes it both stronger and applicable also in the absence of permutational symmetry. The second part of the work is devoted to applications: we provide a single-letter formula for the Stein exponent characterising the discrimination of broad families of null hypotheses vs a composite i.i.d. or an arbitrarily varying alternative hypothesis, and establish a 'constrained de Finetti reduction' statement that covers a wide family of convex constraints. Applications to quantum hypothesis testing are explored in a related paper [Lami, arXiv:today].

Problem

• Extends Chernoff-Stein lemma to composite correlated hypotheses
• Derives Stein exponent for correlated distributions with convexity assumptions
• Establishes single-letter formulas for composite i.i.d. hypothesis testing

Innovation

• Blurring technique applied symbol by symbol
• General Chernoff-Stein lemma for composite correlated hypotheses
• Single-letter formula for Stein exponent in discrimination