When fractional quasi p-norms concentrate

2025-05-26

AI Summary

This paper systematically resolves the long-standing open problem of distance concentration for fractional quasi-p-norms (p ∈ (0,1)) in high-dimensional spaces. Using high-dimensional probability theory, functional inequalities, and distributional perturbation analysis, we establish, for the first time, a p-independent unified exponential concentration bound; rigorously characterize necessary and sufficient conditions for both concentration and anti-concentration; and identify the effective boundary governed by p. We discover extreme instability of concentration behavior over the space of distributions: within any arbitrarily small neighborhood, both concentrating and anti-concentrating distributions coexist. Theoretically, common distributions exhibit strong concentration under this norm, and “optimal p-tuning” fails universally as a mitigation strategy. Furthermore, we construct a family of distributions with tunable concentration rates, enabling the design of novel data encoding paradigms—either concentration-resilient or concentration-enhancing—grounded in principled distributional control.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Concentration of distances in high dimension is an important factor for the development and design of stable and reliable data analysis algorithms. In this paper, we address the fundamental long-standing question about the concentration of distances in high dimension for fractional quasi $p$-norms, $pin(0,1)$. The topic has been at the centre of various theoretical and empirical controversies. Here we, for the first time, identify conditions when fractional quasi $p$-norms concentrate and when they don't. We show that contrary to some earlier suggestions, for broad classes of distributions, fractional quasi $p$-norms admit exponential and uniform in $p$ concentration bounds. For these distributions, the results effectively rule out previously proposed approaches to alleviate concentration by"optimal"setting the values of $p$ in $(0,1)$. At the same time, we specify conditions and the corresponding families of distributions for which one can still control concentration rates by appropriate choices of $p$. We also show that in an arbitrarily small vicinity of a distribution from a large class of distributions for which uniform concentration occurs, there are uncountably many other distributions featuring anti-concentration properties. Importantly, this behavior enables devising relevant data encoding or representation schemes favouring or discouraging distance concentration. The results shed new light on this long-standing problem and resolve the tension around the topic in both theory and empirical evidence reported in the literature.

Problem

• Study distance concentration for fractional quasi p-norms in high dimensions
• Identify conditions for concentration/anti-concentration of fractional quasi p-norms
• Resolve theoretical controversies about p-norm behavior in data analysis

Innovation

• Analyze fractional quasi p-norms concentration conditions
• Identify distributions with controllable concentration rates
• Devise data encoding schemes affecting concentration