Random Feature Models with Learnable Activation Functions

2024-11-29

arXiv.org

AI Summary

Traditional random feature (RF) methods employ fixed activation functions, limiting their generalization capability and task adaptability. This paper proposes the Random Features with Learnable Activation Functions (RFLAF), a novel RF framework wherein the activation function is parameterized and learned end-to-end to enable adaptive representation learning. Key contributions include: (i) the first rigorous kernel-theoretic analysis for RF models incorporating radial basis functions; (ii) a theoretical proof that RFLAF achieves significantly enhanced expressive power using only approximately twice the number of parameters compared to standard RFs; and (iii) the ability to explicitly reconstruct closed-form analytic activation functions from learned weights, thereby unifying high expressivity with interpretability. Theoretical analysis leverages random feature mappings, uniform approximation in the space $C_c(mathbb{R})$, and kernel method principles. Empirical evaluation demonstrates that RFLAF consistently outperforms parameter-matched baselines across diverse tasks, and the learned activation shapes admit clear physical interpretations.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Current random feature models typically rely on fixed activation functions, limiting their ability to capture diverse patterns in data. To address this, we introduce the Random Feature model with Learnable Activation Functions (RFLAF), a novel model that significantly enhances the expressivity and interpretability of traditional random feature (RF) models. We begin by studying the RF model with a single radial basis function, where we discover a new kernel and provide the first theoretical analysis on it. By integrating the basis functions with learnable weights, we show that RFLAF can represent a broad class of random feature models whose activation functions belong in $C_c(mathbb{R})$. Theoretically, we prove that the model requires only about twice the parameter number compared to a traditional RF model to achieve the significant leap in expressivity. Experimentally, RFLAF demonstrates two key advantages: (1) it performs better across various tasks compared to traditional RF model with the same number of parameters, and (2) the optimized weights offer interpretability, as the learned activation function can be directly inferred from these weights. Our model paves the way for developing more expressive and interpretable frameworks within random feature models.

Problem

• Enhancing random feature model adaptability with learnable activation functions
• Expanding function space efficiently using parameterized radial basis functions
• Recovering optimal activation functions directly from data for improved performance

Innovation

• Parameterized activation functions via basis sums
• Radial basis functions for enhanced expressivity
• Learnable activations from data with efficiency