Online Quantile Regression

2024-02-07

arXiv.org

AI Summary

This paper addresses online quantile regression in streaming high-dimensional settings, where feature dimensionality grows with sample size, the time horizon is infinite, and memory is strictly constrained. We propose a stochastic subgradient descent algorithm based on the check loss, equipped with a dynamic step-size schedule to minimize quantile loss incrementally. Theoretical contributions include: (i) the first derivation of an exponential concentration inequality for this setting; (ii) rigorous proof of optimal $O(1/sqrt{t})$ parameter convergence rate and $O(sqrt{T})$ regret bound; and (iii) identification of a strong robustness mechanism—specifically, the step-size strategy effectively suppresses propagation of initial estimation error, rendering long-term performance dependent only on short-term initialization. Extensive simulations demonstrate the method’s stability and significant superiority in high-dimensional dynamic environments.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

This paper addresses the challenge of integrating sequentially arriving data within the quantile regression framework, where the number of features is allowed to grow with the number of observations, the horizon is unknown, and memory is limited. We employ stochastic sub-gradient descent to minimize the empirical check loss and study its statistical properties and regret performance. In our analysis, we unveil the delicate interplay between updating iterates based on individual observations versus batches of observations, revealing distinct regularity properties in each scenario. Our method ensures long-term optimal estimation irrespective of the chosen update strategy. Importantly, our contributions go beyond prior works by achieving exponential-type concentration inequalities and attaining optimal regret and error rates that exhibit only extsf{ short-term} sensitivity to initial errors. A key insight from our study is the delicate statistical analyses and the revelation that appropriate stepsize schemes significantly mitigate the impact of initial errors on subsequent errors and regrets. This underscores the robustness of stochastic sub-gradient descent in handling initial uncertainties, emphasizing its efficacy in scenarios where the sequential arrival of data introduces uncertainties regarding both the horizon and the total number of observations. Additionally, when the initial error rate is well-controlled, there is a trade-off between short-term error rate and long-term optimality. Due to the lack of delicate statistical analysis for squared loss, we also briefly discuss its properties and proper schemes. Extensive simulations support our theoretical findings.

Problem

• Integrating sequential data in quantile regression with growing features
• Achieving optimal regret and error rates with initial error sensitivity
• Balancing short-term error rate and long-term optimality in updates

Innovation

• Stochastic sub-gradient descent for quantile regression
• Exponential-type concentration inequalities achieved
• Stepsize schemes mitigate initial error impact