A Layered Simplex Architecture for Large Alphabets

2026-08-20

AI Summary

本文提出了一种新的贝叶斯估计器，用于解决大字母表下的概率估计问题。该方法通过在概率单纯形上进行分层操作实现，并具有简单构造、可计算的遗憾表达式等优点。

Abstract

Probability estimation over large alphabets under log loss is a well-studied problem, with celebrated methods such as the Good-Turing estimator. We introduce and study a new Bayesian estimator with four notable properties. First, its construction is exceptionally simple: multiply independent uniform draws from the probability simplex coordinate-wise and renormalize. Depth is the only structural parameter, and averaging over depths eliminates the need to tune it. Second, the regret of the resulting mixture, the excess code length it pays relative to a code that knows the source, admits an explicit and efficiently computable expression. Third, despite its simplicity and lack of tuned constants, the estimator is competitive across a diverse set of synthetic and real-text benchmarks with substantially more specialized methods, including Good-Turing. Fourth, the tractability of its regret allows us to identify scaling laws in data, alphabet size, and depth. For Zipf targets with exponent above one, the regret has a simple reading as long as the sample reveals only a small fraction of the alphabet. It closely matches the description length of the set of discovered symbols, at one bit of code per bit of description, plus a further cost per symbol. The data exponent is therefore the rate at which new symbols are discovered.

Problem

• probability estimation
• large alphabets
• log loss
• Bayesian estimator
• regret

Innovation

• Bayesian Estimator
• Probability Simplex
• Regret Computation
• Competitive Performance
• Scaling Laws