On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions

2025-10-17

AI Summary

This work establishes tight generalization bounds for the Random Feature model with Learnable Activation Functions (RFLAF) in regression and classification, focusing on minimizing the required number of random features. To overcome the classical Ω(1/ε²) feature complexity barrier, we propose a data-dependent sampling strategy based on leverage score weighting. This is the first approach to reduce feature complexity to Õ((1/ε)^{1/t}), achieving Ω(1) under favorable conditions—attaining theoretically optimal scaling. Our method integrates learnable activations, random feature mappings, approximate kernel construction, and a unified analysis framework accommodating both MSE and Lipschitz losses. Experiments demonstrate that the weighted RFLAF achieves comparable accuracy with significantly fewer features, validating both theoretical tightness and practical efficacy. The core contribution is the first data-dependent, learnable-activation generalization bound with near-optimal feature complexity, breaking the canonical inverse-square dependence on approximation error ε.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

This paper studies the generalization properties of a recently proposed kernel method, the Random Feature models with Learnable Activation Functions (RFLAF). By applying a data-dependent sampling scheme for generating features, we provide by far the sharpest bounds on the required number of features for learning RFLAF in both the regression and classification tasks. We provide a unified theorem that describes the complexity of the feature number $s$, and discuss the results for the plain sampling scheme and the data-dependent leverage weighted scheme. Through weighted sampling, the bound on $s$ in the MSE loss case is improved from $Ω(1/ε^2)$ to $ ildeΩ((1/ε)^{1/t})$ in general $(tgeq 1)$, and even to $Ω(1)$ when the Gram matrix has a finite rank. For the Lipschitz loss case, the bound is improved from $Ω(1/ε^2)$ to $ ildeΩ((1/ε^2)^{1/t})$. To learn the weighted RFLAF, we also propose an algorithm to find an approximate kernel and then apply the leverage weighted sampling. Empirical results show that the weighted RFLAF achieves the same performances with a significantly fewer number of features compared to the plainly sampled RFLAF, validating our theories and the effectiveness of this method.

Problem

• Analyzing generalization bounds for learnable activation function models
• Improving feature efficiency through data-dependent sampling schemes
• Developing algorithms for approximate kernel learning with weighted sampling

Innovation

• Data-dependent sampling improves feature efficiency
• Leverage weighted sampling reduces required feature count
• Approximate kernel algorithm enables weighted RFLAF learning