Refined Risk Bounds for Unbounded Losses via Transductive Priors

2024-10-29

arXiv.org

AI Summary

This paper investigates risk bounds for sequential linear regression, classification, and logistic regression under unbounded losses in the transductive online learning setting—where the design vector set is known a priori but its ordering is unknown, with no boundedness assumptions on either design vectors or the optimal parameter norm. We propose a horizon-aware transductive prior within the exponential weights framework, replacing traditional ε-cover constructions with sampling from log-concave distributions. Our approach yields, for the first time under unbounded losses, a classification regret bound depending solely on dimension and round count—completely independent of data norms and optimal solution magnitude. For linear regression, we derive a sparsity-aware regret bound that scales with the magnitude of response variables. The algorithm admits polynomial-time approximate implementation. Crucially, our risk bounds are unattainable by conventional denoising methods and are tight in the worst-case adversarial sequence setting.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression, all characterized by unbounded losses in the setup where no assumptions are made on the magnitude of design vectors and the norm of the optimal vector of parameters. The key distinction from existing results lies in our assumption that the set of design vectors is known in advance (though their order is not), a setup sometimes referred to as transductive online learning. While this assumption seems similar to fixed design regression or denoising, we demonstrate that the sequential nature of our algorithms allows us to convert our bounds into statistical ones with random design without making any additional assumptions about the distribution of the design vectors--an impossibility for standard denoising results. Our key tools are based on the exponential weights algorithm with carefully chosen transductive (design-dependent) priors, which exploit the full horizon of the design vectors. Our classification regret bounds have a feature that is only attributed to bounded losses in the literature: they depend solely on the dimension of the parameter space and on the number of rounds, independent of the design vectors or the norm of the optimal solution. For linear regression with squared loss, we further extend our analysis to the sparse case, providing sparsity regret bounds that additionally depend on the magnitude of the response variables. We argue that these improved bounds are specific to the transductive setting and unattainable in the worst-case sequential setup. Our algorithms, in several cases, have polynomial time approximations and reduce to sampling with respect to log-concave measures instead of aggregating over hard-to-construct $varepsilon$-covers of classes.

Problem

• Unbounded losses in sequential learning
• Transductive online learning setup
• Improved bounds with transductive priors

Innovation

• Transductive online learning setup
• Exponential weights algorithm
• Sparsity regret bounds analysis