Adaptive tail index estimation: minimal assumptions and non-asymptotic guarantees

2025-05-28

AI Summary

Adaptive selection of the threshold parameter (k) for the Hill estimator in extreme value theory has long been hindered by reliance on strong second-order regularity assumptions and difficult calibration. Method: Instead of seeking an “optimal (k)”, we pursue a “sufficiently good (k)” with explicit non-asymptotic guarantees. We propose a fully transparent, pre-calibration-free adaptive rule, the first to provide theoretical analysis for fine-grained (k)-grids at practical scales. It avoids bias estimation entirely, relying solely on a variance-type closed-form expression. Our approach integrates adaptive validation ideas from high-dimensional statistics with non-asymptotic probability inequalities, applicable to all regularly varying tails. Contribution/Results: We establish the first non-asymptotic guarantee without any second-order assumptions; under the von Mises condition, convergence attains nearly minimax-optimal rates; simulations demonstrate strong robustness to pathological heavy-tailed distributions.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

A notoriously difficult challenge in extreme value theory is the choice of the number $kll n$, where $n$ is the total sample size, of extreme data points to consider for inference of tail quantities. Existing theoretical guarantees for adaptive methods typically require second-order assumptions or von Mises assumptions that are difficult to verify and often come with tuning parameters that are challenging to calibrate. This paper revisits the problem of adaptive selection of $k$ for the Hill estimator. Our goal is not an `optimal' $k$ but one that is `good enough', in the sense that we strive for non-asymptotic guarantees that might be sub-optimal but are explicit and require minimal conditions. We propose a transparent adaptive rule that does not require preliminary calibration of constants, inspired by `adaptive validation' developed in high-dimensional statistics. A key feature of our approach is the consideration of a grid for $k$ of size $ ll n $, which aligns with common practice among practitioners but has remained unexplored in theoretical analysis. Our rule only involves an explicit expression of a variance-type term; in particular, it does not require controlling or estimating a biasterm. Our theoretical analysis is valid for all heavy-tailed distributions, specifically for all regularly varying survival functions. Furthermore, when von Mises conditions hold, our method achieves `almost' minimax optimality with a rate of $sqrt{log log n}~ n^{-| ho|/(1+2| ho|)}$ when the grid size is of order $log n$, in contrast to the $ (log log (n)/n)^{| ho|/(1+2| ho|)} $ rate in existing work. Our simulations show that our approach performs particularly well for ill-behaved distributions.

Problem

• Adaptive selection of k for Hill estimator
• Non-asymptotic guarantees with minimal conditions
• Handling heavy-tailed distributions without bias estimation

Innovation

• Adaptive rule without preliminary calibration
• Grid for k of size << n
• Non-asymptotic guarantees for heavy tails